This is a HTML-ification of Workflow state (workflow.state) and
the associated files in the workflow output directory.
Below is the doc section of the workflow (.workflow file).
Produce, transform and plot fio-output ====================================== This workflow assumes that the 'prep.workflow' has completed. The comparison and plots consists of * bandwidth in KiB / seconds * latency is mean and in nano-seconds * IOPS are 'raw' / base-unit The above in a bar-plot at iodepth=1, and a line-plot as a function of iodepth. See Artifacts for the generated plots
If a workflow step (worklet / run) or test produced any artifacts, then they are listed here.
This is the configuration (.config file) with any
variable-placeholders such as {{ local.env.HOME }} filled out.
devices:
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme0n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme1n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme2n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme3n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme4n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme5n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme6n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme7n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng0n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng1n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng2n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng3n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng4n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng5n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng6n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng7n1
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:41:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:42:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:43:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:44:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:45:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:46:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:47:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:48:00.0'
fio:
bin: /opt/fio/bin/fio
build:
prefix: /opt/fio
engines:
io_uring:
type: builtin
io_uring_cmd:
type: builtin
libaio:
type: builtin
spdk_bdev:
path: /root/git/spdk/build/fio/spdk_bdev
type: external_preload
spdk_nvme:
path: /root/git/spdk/build/fio/spdk_nvme
type: external_preload
xnvme:
path: /usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so
type: external_dynamic
repository:
path: /root/git/fio
remote: https://github.com/axboe/fio.git
tag: fio-3.32
liburing:
repository:
path: /root/git/liburing
remote: https://github.com/axboe/liburing.git
tag: liburing-2.2
os:
name: debian
version: bullseye
spdk:
build:
prefix: /opt/spdk
repository:
branch: for_bdev_xnvme_v6
path: /root/git/spdk
remote: https://github.com/safl/spdk.git
transport:
ssh:
hostname: fabio
port: 22
username: root
xnvme:
build:
type: debug
repository:
path: /root/git/xnvme
remote: https://github.com/OpenMPDK/xNVMe.git
tag: v0.6.0
Steps uses worklets. A description of the worklet used by
the current step is provided below. The description is extracted from the docstring
of the worklet and provided here in verbatim / as-is.
Run fio...
{'description': '\n Run fio...\n',
'extras': {'runlog': {'cmd_01': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_01.output'),
'state': {'begin': 1666724735.1971772,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8816070556640625,
'end': 1666724736.0787842,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_01.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_01.state')},
'cmd_02': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'96312.06 IOPS 376.22 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'96312.06 IOPS 376.22 MiB/s\n'
'[2022-10-25 23:03:47.429048] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:03:47.429120] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:03:47.429132] bdevperf '
'[2022-10-25 23:03:47.429142] '
'--no-shconf [2022-10-25 '
'23:03:47.429151] -l 0 [2022-10-25 '
'23:03:47.429169] --huge-unlink '
'[2022-10-25 23:03:47.429179] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:03:47.429188] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:03:47.429196] '
'--log-level=user1:6 [2022-10-25 '
'23:03:47.429205] --iova-mode=pa '
'[2022-10-25 23:03:47.429219] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:03:47.429227] '
'--match-allocations [2022-10-25 '
'23:03:47.429235] '
'--file-prefix=spdk_pid82863 '
'[2022-10-25 23:03:47.429243] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:03:47.567686] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:03:47.609539] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:03:47.610229] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_02.output'),
'state': {'begin': 1666724737.1061788,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.127448797225952,
'end': 1666724748.2336276,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_02.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_02.state')},
'cmd_03': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_03.output'),
'state': {'begin': 1666724748.236189,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.789337158203125,
'end': 1666724749.025526,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_03.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_03.state')},
'cmd_04': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'91332.77 IOPS 356.77 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'91332.77 IOPS 356.77 MiB/s\n'
'[2022-10-25 23:04:00.370178] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:04:00.370248] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:04:00.370259] bdevperf '
'[2022-10-25 23:04:00.370268] '
'--no-shconf [2022-10-25 '
'23:04:00.370277] -l 0 [2022-10-25 '
'23:04:00.370285] --huge-unlink '
'[2022-10-25 23:04:00.370293] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:04:00.370302] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:04:00.370309] '
'--log-level=user1:6 [2022-10-25 '
'23:04:00.370317] --iova-mode=pa '
'[2022-10-25 23:04:00.370325] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:04:00.370333] '
'--match-allocations [2022-10-25 '
'23:04:00.370341] '
'--file-prefix=spdk_pid82888 '
'[2022-10-25 23:04:00.370350] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:04:00.508790] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:04:00.557574] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:04:00.558236] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_04.output'),
'state': {'begin': 1666724750.0590663,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.12565016746521,
'end': 1666724761.1847165,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_04.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_04.state')},
'cmd_05': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_05.output'),
'state': {'begin': 1666724761.1874182,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7818958759307861,
'end': 1666724761.969314,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_05.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_05.state')},
'cmd_06': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'91051.09 IOPS 355.67 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'91051.09 IOPS 355.67 MiB/s\n'
'[2022-10-25 23:04:13.314791] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:04:13.314863] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:04:13.314874] bdevperf '
'[2022-10-25 23:04:13.314882] '
'--no-shconf [2022-10-25 '
'23:04:13.314890] -l 0 [2022-10-25 '
'23:04:13.314898] --huge-unlink '
'[2022-10-25 23:04:13.314905] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:04:13.314913] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:04:13.314920] '
'--log-level=user1:6 [2022-10-25 '
'23:04:13.314927] --iova-mode=pa '
'[2022-10-25 23:04:13.314935] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:04:13.314942] '
'--match-allocations [2022-10-25 '
'23:04:13.314949] '
'--file-prefix=spdk_pid82913 '
'[2022-10-25 23:04:13.314956] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:04:13.453309] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:04:13.493561] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:04:13.494240] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_06.output'),
'state': {'begin': 1666724762.9952679,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.11287784576416,
'end': 1666724774.1081457,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_06.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_06.state')},
'cmd_07': {'output': 'rm: cannot remove '
"'/tmp/spdk_bdev_uring_foo.conf': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_07.output'),
'state': {'begin': 1666724774.1112564,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7770850658416748,
'end': 1666724774.8883414,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_07.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_07.state')},
'cmd_08': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'87268.14 IOPS 340.89 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'87268.14 IOPS 340.89 MiB/s\n'
'[2022-10-25 23:04:26.232081] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:04:26.232151] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:04:26.232162] bdevperf '
'[2022-10-25 23:04:26.232171] '
'--no-shconf [2022-10-25 '
'23:04:26.232180] -l 0 [2022-10-25 '
'23:04:26.232189] --huge-unlink '
'[2022-10-25 23:04:26.232197] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:04:26.232205] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:04:26.232212] '
'--log-level=user1:6 [2022-10-25 '
'23:04:26.232221] --iova-mode=pa '
'[2022-10-25 23:04:26.232229] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:04:26.232237] '
'--match-allocations [2022-10-25 '
'23:04:26.232245] '
'--file-prefix=spdk_pid82940 '
'[2022-10-25 23:04:26.232253] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:04:26.370496] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:04:26.425602] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:04:26.426289] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_08.output'),
'state': {'begin': 1666724775.910652,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.134417533874512,
'end': 1666724787.0450695,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_08.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_08.state')},
'cmd_09': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_09.output'),
'state': {'begin': 1666724787.048074,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7892162799835205,
'end': 1666724787.8372903,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_09.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_09.state')},
'cmd_10': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'87338.70 IOPS 341.17 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'87338.70 IOPS 341.17 MiB/s\n'
'[2022-10-25 23:04:39.195818] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:04:39.195889] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:04:39.195900] bdevperf '
'[2022-10-25 23:04:39.195909] '
'--no-shconf [2022-10-25 '
'23:04:39.195918] -l 0 [2022-10-25 '
'23:04:39.195926] --huge-unlink '
'[2022-10-25 23:04:39.195934] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:04:39.195942] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:04:39.195949] '
'--log-level=user1:6 [2022-10-25 '
'23:04:39.195957] --iova-mode=pa '
'[2022-10-25 23:04:39.195966] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:04:39.195973] '
'--match-allocations [2022-10-25 '
'23:04:39.195981] '
'--file-prefix=spdk_pid82965 '
'[2022-10-25 23:04:39.195989] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:04:39.333951] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:04:39.381531] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:04:39.382192] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_10.output'),
'state': {'begin': 1666724788.8770678,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.116129159927368,
'end': 1666724799.993197,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_10.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_10.state')},
'cmd_100': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'223720.83 IOPS 873.91 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'223720.83 IOPS 873.91 MiB/s\n'
'[2022-10-25 23:20:50.653460] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:20:50.653531] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:20:50.653542] bdevperf '
'[2022-10-25 23:20:50.653551] '
'--no-shconf [2022-10-25 '
'23:20:50.653560] -l 0 '
'[2022-10-25 23:20:50.653567] '
'--huge-unlink [2022-10-25 '
'23:20:50.653575] '
'--log-level=lib.eal:6 '
'[2022-10-25 23:20:50.653583] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:20:50.653591] '
'--log-level=user1:6 [2022-10-25 '
'23:20:50.653599] --iova-mode=pa '
'[2022-10-25 23:20:50.653607] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:20:50.653614] '
'--match-allocations [2022-10-25 '
'23:20:50.653622] '
'--file-prefix=spdk_pid84111 '
'[2022-10-25 23:20:50.653629] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:20:50.792163] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:20:50.833572] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-25 23:20:50.834238] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_100.output'),
'state': {'begin': 1666725760.3471627,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.106633186340332,
'end': 1666725771.453796,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_100.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_100.state')},
'cmd_101': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_101.output'),
'state': {'begin': 1666725771.4565046,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8669767379760742,
'end': 1666725772.3234813,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_101.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_101.state')},
'cmd_102': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'223708.71 IOPS 873.86 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'223708.71 IOPS 873.86 MiB/s\n'
'[2022-10-25 23:21:03.647189] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:21:03.647258] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:21:03.647268] bdevperf '
'[2022-10-25 23:21:03.647277] '
'--no-shconf [2022-10-25 '
'23:21:03.647286] -l 0 '
'[2022-10-25 23:21:03.647295] '
'--huge-unlink [2022-10-25 '
'23:21:03.647303] '
'--log-level=lib.eal:6 '
'[2022-10-25 23:21:03.647311] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:21:03.647318] '
'--log-level=user1:6 [2022-10-25 '
'23:21:03.647326] --iova-mode=pa '
'[2022-10-25 23:21:03.647334] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:21:03.647342] '
'--match-allocations [2022-10-25 '
'23:21:03.647350] '
'--file-prefix=spdk_pid84137 '
'[2022-10-25 23:21:03.647359] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:21:03.785565] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:21:03.829482] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-25 23:21:03.830158] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_102.output'),
'state': {'begin': 1666725773.3505797,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.105631828308105,
'end': 1666725784.4562116,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_102.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_102.state')},
'cmd_103': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_103.output'),
'state': {'begin': 1666725784.4601681,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7901983261108398,
'end': 1666725785.2503664,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_103.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_103.state')},
'cmd_104': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'223714.32 IOPS 873.88 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'223714.32 IOPS 873.88 MiB/s\n'
'[2022-10-25 23:21:16.578379] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:21:16.578446] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:21:16.578457] bdevperf '
'[2022-10-25 23:21:16.578465] '
'--no-shconf [2022-10-25 '
'23:21:16.578474] -l 0 '
'[2022-10-25 23:21:16.578482] '
'--huge-unlink [2022-10-25 '
'23:21:16.578489] '
'--log-level=lib.eal:6 '
'[2022-10-25 23:21:16.578497] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:21:16.578504] '
'--log-level=user1:6 [2022-10-25 '
'23:21:16.578512] --iova-mode=pa '
'[2022-10-25 23:21:16.578519] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:21:16.578526] '
'--match-allocations [2022-10-25 '
'23:21:16.578534] '
'--file-prefix=spdk_pid84162 '
'[2022-10-25 23:21:16.578541] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:21:16.717093] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:21:16.765505] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-25 23:21:16.766176] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_104.output'),
'state': {'begin': 1666725786.274423,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.119341611862183,
'end': 1666725797.3937645,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_104.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_104.state')},
'cmd_105': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_105.output'),
'state': {'begin': 1666725797.3967102,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7889127731323242,
'end': 1666725798.185623,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_105.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_105.state')},
'cmd_106': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'223656.87 IOPS 873.66 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'223656.87 IOPS 873.66 MiB/s\n'
'[2022-10-25 23:21:29.521276] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:21:29.521350] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:21:29.521362] bdevperf '
'[2022-10-25 23:21:29.521371] '
'--no-shconf [2022-10-25 '
'23:21:29.521380] -l 0 '
'[2022-10-25 23:21:29.521388] '
'--huge-unlink [2022-10-25 '
'23:21:29.521396] '
'--log-level=lib.eal:6 '
'[2022-10-25 23:21:29.521404] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:21:29.521412] '
'--log-level=user1:6 [2022-10-25 '
'23:21:29.521420] --iova-mode=pa '
'[2022-10-25 23:21:29.521428] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:21:29.521436] '
'--match-allocations [2022-10-25 '
'23:21:29.521443] '
'--file-prefix=spdk_pid84187 '
'[2022-10-25 23:21:29.521454] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:21:29.660283] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:21:29.701545] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-25 23:21:29.702222] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_106.output'),
'state': {'begin': 1666725799.2229874,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.107008457183838,
'end': 1666725810.3299959,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_106.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_106.state')},
'cmd_107': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_107.output'),
'state': {'begin': 1666725810.333197,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7849862575531006,
'end': 1666725811.1181834,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_107.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_107.state')},
'cmd_108': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'223746.27 IOPS 874.01 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'223746.27 IOPS 874.01 MiB/s\n'
'[2022-10-25 23:21:42.426968] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:21:42.427037] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:21:42.427047] bdevperf '
'[2022-10-25 23:21:42.427056] '
'--no-shconf [2022-10-25 '
'23:21:42.427063] -l 0 '
'[2022-10-25 23:21:42.427071] '
'--huge-unlink [2022-10-25 '
'23:21:42.427078] '
'--log-level=lib.eal:6 '
'[2022-10-25 23:21:42.427086] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:21:42.427093] '
'--log-level=user1:6 [2022-10-25 '
'23:21:42.427100] --iova-mode=pa '
'[2022-10-25 23:21:42.427108] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:21:42.427125] '
'--match-allocations [2022-10-25 '
'23:21:42.427132] '
'--file-prefix=spdk_pid84213 '
'[2022-10-25 23:21:42.427139] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:21:42.565813] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:21:42.613477] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-25 23:21:42.614155] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_108.output'),
'state': {'begin': 1666725812.1362538,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.110110998153687,
'end': 1666725823.2463648,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_108.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_108.state')},
'cmd_109': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_109.output'),
'state': {'begin': 1666725823.249396,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.781137228012085,
'end': 1666725824.0305333,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_109.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_109.state')},
'cmd_11': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_11.output'),
'state': {'begin': 1666724799.9959278,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.79339599609375,
'end': 1666724800.7893238,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_11.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_11.state')},
'cmd_110': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'223741.17 IOPS 873.99 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'223741.17 IOPS 873.99 MiB/s\n'
'[2022-10-25 23:21:55.355943] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:21:55.356022] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:21:55.356033] bdevperf '
'[2022-10-25 23:21:55.356042] '
'--no-shconf [2022-10-25 '
'23:21:55.356050] -l 0 '
'[2022-10-25 23:21:55.356057] '
'--huge-unlink [2022-10-25 '
'23:21:55.356065] '
'--log-level=lib.eal:6 '
'[2022-10-25 23:21:55.356072] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:21:55.356079] '
'--log-level=user1:6 [2022-10-25 '
'23:21:55.356087] --iova-mode=pa '
'[2022-10-25 23:21:55.356094] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:21:55.356102] '
'--match-allocations [2022-10-25 '
'23:21:55.356109] '
'--file-prefix=spdk_pid84238 '
'[2022-10-25 23:21:55.356116] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:21:55.494518] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:21:55.549627] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-25 23:21:55.550308] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_110.output'),
'state': {'begin': 1666725825.0630252,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.172744035720825,
'end': 1666725836.2357693,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_110.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_110.state')},
'cmd_111': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_111.output'),
'state': {'begin': 1666725836.238611,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7857277393341064,
'end': 1666725837.0243387,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_111.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_111.state')},
'cmd_112': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'223785.38 IOPS 874.16 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'223785.38 IOPS 874.16 MiB/s\n'
'[2022-10-25 23:22:08.359371] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:22:08.359442] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:22:08.359452] bdevperf '
'[2022-10-25 23:22:08.359461] '
'--no-shconf [2022-10-25 '
'23:22:08.359470] -l 0 '
'[2022-10-25 23:22:08.359478] '
'--huge-unlink [2022-10-25 '
'23:22:08.359486] '
'--log-level=lib.eal:6 '
'[2022-10-25 23:22:08.359494] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:22:08.359501] '
'--log-level=user1:6 [2022-10-25 '
'23:22:08.359511] --iova-mode=pa '
'[2022-10-25 23:22:08.359519] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:22:08.359527] '
'--match-allocations [2022-10-25 '
'23:22:08.359535] '
'--file-prefix=spdk_pid84265 '
'[2022-10-25 23:22:08.359542] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:22:08.497812] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:22:08.545510] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-25 23:22:08.546171] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_112.output'),
'state': {'begin': 1666725838.058469,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.185937643051147,
'end': 1666725849.2444067,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_112.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_112.state')},
'cmd_113': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_113.output'),
'state': {'begin': 1666725849.247445,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7806620597839355,
'end': 1666725850.0281072,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_113.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_113.state')},
'cmd_114': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'223763.55 IOPS 874.08 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'223763.55 IOPS 874.08 MiB/s\n'
'[2022-10-25 23:22:21.359973] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:22:21.360044] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:22:21.360054] bdevperf '
'[2022-10-25 23:22:21.360063] '
'--no-shconf [2022-10-25 '
'23:22:21.360071] -l 0 '
'[2022-10-25 23:22:21.360079] '
'--huge-unlink [2022-10-25 '
'23:22:21.360087] '
'--log-level=lib.eal:6 '
'[2022-10-25 23:22:21.360095] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:22:21.360103] '
'--log-level=user1:6 [2022-10-25 '
'23:22:21.360110] --iova-mode=pa '
'[2022-10-25 23:22:21.360118] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:22:21.360126] '
'--match-allocations [2022-10-25 '
'23:22:21.360134] '
'--file-prefix=spdk_pid84290 '
'[2022-10-25 23:22:21.360141] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:22:21.498321] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:22:21.545611] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-25 23:22:21.546279] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_114.output'),
'state': {'begin': 1666725851.0596964,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.172929048538208,
'end': 1666725862.2326255,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_114.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_114.state')},
'cmd_115': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_115.output'),
'state': {'begin': 1666725862.2352889,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7849881649017334,
'end': 1666725863.020277,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_115.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_115.state')},
'cmd_116': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'223690.52 IOPS 873.79 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'223690.52 IOPS 873.79 MiB/s\n'
'[2022-10-25 23:22:34.349231] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:22:34.349305] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:22:34.349316] bdevperf '
'[2022-10-25 23:22:34.349325] '
'--no-shconf [2022-10-25 '
'23:22:34.349333] -l 0 '
'[2022-10-25 23:22:34.349342] '
'--huge-unlink [2022-10-25 '
'23:22:34.349350] '
'--log-level=lib.eal:6 '
'[2022-10-25 23:22:34.349358] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:22:34.349365] '
'--log-level=user1:6 [2022-10-25 '
'23:22:34.349374] --iova-mode=pa '
'[2022-10-25 23:22:34.349383] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:22:34.349391] '
'--match-allocations [2022-10-25 '
'23:22:34.349399] '
'--file-prefix=spdk_pid84315 '
'[2022-10-25 23:22:34.349407] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:22:34.487798] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:22:34.529488] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-25 23:22:34.530150] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_116.output'),
'state': {'begin': 1666725864.0558655,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.188930749893188,
'end': 1666725875.2447963,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_116.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_116.state')},
'cmd_117': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_117.output'),
'state': {'begin': 1666725875.2476728,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8118739128112793,
'end': 1666725876.0595467,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_117.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_117.state')},
'cmd_118': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'223783.71 IOPS 874.16 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'223783.71 IOPS 874.16 MiB/s\n'
'[2022-10-25 23:22:47.374691] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:22:47.374761] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:22:47.374772] bdevperf '
'[2022-10-25 23:22:47.374781] '
'--no-shconf [2022-10-25 '
'23:22:47.374789] -l 0 '
'[2022-10-25 23:22:47.374797] '
'--huge-unlink [2022-10-25 '
'23:22:47.374806] '
'--log-level=lib.eal:6 '
'[2022-10-25 23:22:47.374814] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:22:47.374821] '
'--log-level=user1:6 [2022-10-25 '
'23:22:47.374847] --iova-mode=pa '
'[2022-10-25 23:22:47.374857] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:22:47.374866] '
'--match-allocations [2022-10-25 '
'23:22:47.374873] '
'--file-prefix=spdk_pid84340 '
'[2022-10-25 23:22:47.374887] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:22:47.513230] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:22:47.553617] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-25 23:22:47.554293] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_118.output'),
'state': {'begin': 1666725877.0861018,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.163384199142456,
'end': 1666725888.249486,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_118.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_118.state')},
'cmd_119': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_119.output'),
'state': {'begin': 1666725888.2520986,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7791037559509277,
'end': 1666725889.0312023,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_119.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_119.state')},
'cmd_12': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'87084.15 IOPS 340.17 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'87084.15 IOPS 340.17 MiB/s\n'
'[2022-10-25 23:04:52.128230] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:04:52.128295] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:04:52.128306] bdevperf '
'[2022-10-25 23:04:52.128315] '
'--no-shconf [2022-10-25 '
'23:04:52.128323] -l 0 [2022-10-25 '
'23:04:52.128331] --huge-unlink '
'[2022-10-25 23:04:52.128339] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:04:52.128347] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:04:52.128355] '
'--log-level=user1:6 [2022-10-25 '
'23:04:52.128362] --iova-mode=pa '
'[2022-10-25 23:04:52.128371] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:04:52.128379] '
'--match-allocations [2022-10-25 '
'23:04:52.128386] '
'--file-prefix=spdk_pid82990 '
'[2022-10-25 23:04:52.128394] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:04:52.266924] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:04:52.325526] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:04:52.326204] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_12.output'),
'state': {'begin': 1666724801.8169408,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.112801551818848,
'end': 1666724812.9297423,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_12.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_12.state')},
'cmd_120': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'223745.85 IOPS 874.01 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'223745.85 IOPS 874.01 MiB/s\n'
'[2022-10-25 23:23:00.358689] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:23:00.358758] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:23:00.358768] bdevperf '
'[2022-10-25 23:23:00.358777] '
'--no-shconf [2022-10-25 '
'23:23:00.358786] -l 0 '
'[2022-10-25 23:23:00.358795] '
'--huge-unlink [2022-10-25 '
'23:23:00.358803] '
'--log-level=lib.eal:6 '
'[2022-10-25 23:23:00.358811] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:23:00.358819] '
'--log-level=user1:6 [2022-10-25 '
'23:23:00.358826] --iova-mode=pa '
'[2022-10-25 23:23:00.358834] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:23:00.358842] '
'--match-allocations [2022-10-25 '
'23:23:00.358850] '
'--file-prefix=spdk_pid84365 '
'[2022-10-25 23:23:00.358858] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:23:00.497334] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:23:00.545520] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-25 23:23:00.546189] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_120.output'),
'state': {'begin': 1666725890.067811,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.190223217010498,
'end': 1666725901.2580342,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_120.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_120.state')},
'cmd_13': {'output': 'rm: cannot remove '
"'/tmp/spdk_bdev_xnvme_io_uring.conf': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_13.output'),
'state': {'begin': 1666724812.9327962,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7896976470947266,
'end': 1666724813.722494,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_13.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_13.state')},
'cmd_14': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'95399.03 IOPS 372.65 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'95399.03 IOPS 372.65 MiB/s\n'
'[2022-10-25 23:05:05.060149] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:05:05.060219] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:05:05.060231] bdevperf '
'[2022-10-25 23:05:05.060240] '
'--no-shconf [2022-10-25 '
'23:05:05.060249] -l 0 [2022-10-25 '
'23:05:05.060257] --huge-unlink '
'[2022-10-25 23:05:05.060265] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:05:05.060273] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:05:05.060280] '
'--log-level=user1:6 [2022-10-25 '
'23:05:05.060288] --iova-mode=pa '
'[2022-10-25 23:05:05.060296] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:05:05.060304] '
'--match-allocations [2022-10-25 '
'23:05:05.060312] '
'--file-prefix=spdk_pid83016 '
'[2022-10-25 23:05:05.060321] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:05:05.198531] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:05:05.245527] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:05:05.246196] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_14.output'),
'state': {'begin': 1666724814.740206,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.111697912216187,
'end': 1666724825.851904,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_14.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_14.state')},
'cmd_15': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_15.output'),
'state': {'begin': 1666724825.8548682,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7805404663085938,
'end': 1666724826.6354086,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_15.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_15.state')},
'cmd_16': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'95115.58 IOPS 371.55 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'95115.58 IOPS 371.55 MiB/s\n'
'[2022-10-25 23:05:17.970774] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:05:17.970842] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:05:17.970852] bdevperf '
'[2022-10-25 23:05:17.970861] '
'--no-shconf [2022-10-25 '
'23:05:17.970869] -l 0 [2022-10-25 '
'23:05:17.970877] --huge-unlink '
'[2022-10-25 23:05:17.970886] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:05:17.970894] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:05:17.970902] '
'--log-level=user1:6 [2022-10-25 '
'23:05:17.970910] --iova-mode=pa '
'[2022-10-25 23:05:17.970918] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:05:17.970926] '
'--match-allocations [2022-10-25 '
'23:05:17.970934] '
'--file-prefix=spdk_pid83041 '
'[2022-10-25 23:05:17.970943] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:05:18.109398] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:05:18.153610] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:05:18.154319] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_16.output'),
'state': {'begin': 1666724827.6594691,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.105032682418823,
'end': 1666724838.7645018,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_16.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_16.state')},
'cmd_17': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_17.output'),
'state': {'begin': 1666724838.7677443,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7861323356628418,
'end': 1666724839.5538766,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_17.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_17.state')},
'cmd_18': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'95206.38 IOPS 371.90 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'95206.38 IOPS 371.90 MiB/s\n'
'[2022-10-25 23:05:30.898804] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:05:30.898875] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:05:30.898887] bdevperf '
'[2022-10-25 23:05:30.898896] '
'--no-shconf [2022-10-25 '
'23:05:30.898905] -l 0 [2022-10-25 '
'23:05:30.898912] --huge-unlink '
'[2022-10-25 23:05:30.898921] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:05:30.898929] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:05:30.898937] '
'--log-level=user1:6 [2022-10-25 '
'23:05:30.898945] --iova-mode=pa '
'[2022-10-25 23:05:30.898953] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:05:30.898960] '
'--match-allocations [2022-10-25 '
'23:05:30.898968] '
'--file-prefix=spdk_pid83066 '
'[2022-10-25 23:05:30.898976] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:05:31.037583] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:05:31.081613] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:05:31.082310] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_18.output'),
'state': {'begin': 1666724840.5913122,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.099512338638306,
'end': 1666724851.6908245,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_18.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_18.state')},
'cmd_19': {'output': 'rm: cannot remove '
"'/tmp/spdk_bdev_aio_foo.conf': No "
'such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_19.output'),
'state': {'begin': 1666724851.6936128,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7899956703186035,
'end': 1666724852.4836085,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_19.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_19.state')},
'cmd_20': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'94374.27 IOPS 368.65 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'94374.27 IOPS 368.65 MiB/s\n'
'[2022-10-25 23:05:43.831772] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:05:43.831843] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:05:43.831854] bdevperf '
'[2022-10-25 23:05:43.831863] '
'--no-shconf [2022-10-25 '
'23:05:43.831872] -l 0 [2022-10-25 '
'23:05:43.831880] --huge-unlink '
'[2022-10-25 23:05:43.831888] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:05:43.831897] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:05:43.831905] '
'--log-level=user1:6 [2022-10-25 '
'23:05:43.831913] --iova-mode=pa '
'[2022-10-25 23:05:43.831921] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:05:43.831929] '
'--match-allocations [2022-10-25 '
'23:05:43.831937] '
'--file-prefix=spdk_pid83091 '
'[2022-10-25 23:05:43.831946] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:05:43.970469] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:05:44.009497] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:05:44.010185] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_20.output'),
'state': {'begin': 1666724853.5132108,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.154639959335327,
'end': 1666724864.6678507,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_20.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_20.state')},
'cmd_21': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_21.output'),
'state': {'begin': 1666724864.6707578,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7889523506164551,
'end': 1666724865.4597101,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_21.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_21.state')},
'cmd_22': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'94300.13 IOPS 368.36 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'94300.13 IOPS 368.36 MiB/s\n'
'[2022-10-25 23:05:56.804496] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:05:56.804569] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:05:56.804581] bdevperf '
'[2022-10-25 23:05:56.804590] '
'--no-shconf [2022-10-25 '
'23:05:56.804599] -l 0 [2022-10-25 '
'23:05:56.804608] --huge-unlink '
'[2022-10-25 23:05:56.804617] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:05:56.804625] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:05:56.804634] '
'--log-level=user1:6 [2022-10-25 '
'23:05:56.804643] --iova-mode=pa '
'[2022-10-25 23:05:56.804652] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:05:56.804660] '
'--match-allocations [2022-10-25 '
'23:05:56.804670] '
'--file-prefix=spdk_pid83116 '
'[2022-10-25 23:05:56.804679] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:05:56.943818] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:05:56.989578] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:05:56.990266] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_22.output'),
'state': {'begin': 1666724866.492729,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.173961639404297,
'end': 1666724877.6666906,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_22.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_22.state')},
'cmd_23': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_23.output'),
'state': {'begin': 1666724877.6687572,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7861847877502441,
'end': 1666724878.454942,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_23.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_23.state')},
'cmd_24': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'94145.55 IOPS 367.76 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'94145.55 IOPS 367.76 MiB/s\n'
'[2022-10-25 23:06:09.798113] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:06:09.798184] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:06:09.798194] bdevperf '
'[2022-10-25 23:06:09.798203] '
'--no-shconf [2022-10-25 '
'23:06:09.798211] -l 0 [2022-10-25 '
'23:06:09.798219] --huge-unlink '
'[2022-10-25 23:06:09.798228] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:06:09.798235] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:06:09.798243] '
'--log-level=user1:6 [2022-10-25 '
'23:06:09.798251] --iova-mode=pa '
'[2022-10-25 23:06:09.798259] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:06:09.798267] '
'--match-allocations [2022-10-25 '
'23:06:09.798274] '
'--file-prefix=spdk_pid83141 '
'[2022-10-25 23:06:09.798282] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:06:09.937122] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:06:09.989458] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:06:09.990130] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_24.output'),
'state': {'begin': 1666724879.4857738,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.173627376556396,
'end': 1666724890.6594012,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_24.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_24.state')},
'cmd_25': {'output': 'rm: cannot remove '
"'/tmp/spdk_bdev_xnvme_libaio.conf': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_25.output'),
'state': {'begin': 1666724890.6621559,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7881510257720947,
'end': 1666724891.450307,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_25.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_25.state')},
'cmd_26': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'92010.24 IOPS 359.42 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'92010.24 IOPS 359.42 MiB/s\n'
'[2022-10-25 23:06:22.801816] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:06:22.801884] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:06:22.801895] bdevperf '
'[2022-10-25 23:06:22.801904] '
'--no-shconf [2022-10-25 '
'23:06:22.801913] -l 0 [2022-10-25 '
'23:06:22.801920] --huge-unlink '
'[2022-10-25 23:06:22.801928] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:06:22.801936] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:06:22.801943] '
'--log-level=user1:6 [2022-10-25 '
'23:06:22.801968] --iova-mode=pa '
'[2022-10-25 23:06:22.801976] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:06:22.801983] '
'--match-allocations [2022-10-25 '
'23:06:22.801990] '
'--file-prefix=spdk_pid83166 '
'[2022-10-25 23:06:22.802006] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:06:22.940465] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:06:22.981534] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:06:22.982224] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_26.output'),
'state': {'begin': 1666724892.4849267,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.1791250705719,
'end': 1666724903.6640518,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_26.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_26.state')},
'cmd_27': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_27.output'),
'state': {'begin': 1666724903.667292,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7869460582733154,
'end': 1666724904.4542382,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_27.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_27.state')},
'cmd_28': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'92095.63 IOPS 359.75 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'92095.63 IOPS 359.75 MiB/s\n'
'[2022-10-25 23:06:35.798932] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:06:35.799005] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:06:35.799016] bdevperf '
'[2022-10-25 23:06:35.799025] '
'--no-shconf [2022-10-25 '
'23:06:35.799033] -l 0 [2022-10-25 '
'23:06:35.799041] --huge-unlink '
'[2022-10-25 23:06:35.799050] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:06:35.799058] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:06:35.799065] '
'--log-level=user1:6 [2022-10-25 '
'23:06:35.799075] --iova-mode=pa '
'[2022-10-25 23:06:35.799082] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:06:35.799091] '
'--match-allocations [2022-10-25 '
'23:06:35.799099] '
'--file-prefix=spdk_pid83193 '
'[2022-10-25 23:06:35.799107] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:06:35.937685] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:06:35.993482] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:06:35.994154] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_28.output'),
'state': {'begin': 1666724905.4874675,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.17349100112915,
'end': 1666724916.6609585,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_28.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_28.state')},
'cmd_29': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_29.output'),
'state': {'begin': 1666724916.6637738,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7949099540710449,
'end': 1666724917.4586837,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_29.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_29.state')},
'cmd_30': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'91701.04 IOPS 358.21 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'91701.04 IOPS 358.21 MiB/s\n'
'[2022-10-25 23:06:48.806875] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:06:48.806943] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:06:48.806953] bdevperf '
'[2022-10-25 23:06:48.806962] '
'--no-shconf [2022-10-25 '
'23:06:48.806970] -l 0 [2022-10-25 '
'23:06:48.806977] --huge-unlink '
'[2022-10-25 23:06:48.806985] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:06:48.806993] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:06:48.807000] '
'--log-level=user1:6 [2022-10-25 '
'23:06:48.807025] --iova-mode=pa '
'[2022-10-25 23:06:48.807033] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:06:48.807041] '
'--match-allocations [2022-10-25 '
'23:06:48.807049] '
'--file-prefix=spdk_pid83218 '
'[2022-10-25 23:06:48.807064] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:06:48.945250] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:06:48.985598] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:06:48.986291] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_30.output'),
'state': {'begin': 1666724918.4907243,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 1 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.182631492614746,
'end': 1666724929.6733558,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_30.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_30.state')},
'cmd_31': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_31.output'),
'state': {'begin': 1666724929.6761372,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7932009696960449,
'end': 1666724930.4693382,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_31.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_31.state')},
'cmd_32': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'137677.93 IOPS 537.80 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'137677.93 IOPS 537.80 MiB/s\n'
'[2022-10-25 23:07:01.817511] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:07:01.817582] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:07:01.817593] bdevperf '
'[2022-10-25 23:07:01.817602] '
'--no-shconf [2022-10-25 '
'23:07:01.817611] -l 0 [2022-10-25 '
'23:07:01.817618] --huge-unlink '
'[2022-10-25 23:07:01.817628] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:07:01.817635] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:07:01.817643] '
'--log-level=user1:6 [2022-10-25 '
'23:07:01.817651] --iova-mode=pa '
'[2022-10-25 23:07:01.817659] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:07:01.817667] '
'--match-allocations [2022-10-25 '
'23:07:01.817675] '
'--file-prefix=spdk_pid83243 '
'[2022-10-25 23:07:01.817684] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:07:01.956216] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:07:02.001505] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:07:02.002177] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_32.output'),
'state': {'begin': 1666724931.5049977,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.115170955657959,
'end': 1666724942.6201687,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_32.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_32.state')},
'cmd_33': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_33.output'),
'state': {'begin': 1666724942.6232758,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7770218849182129,
'end': 1666724943.4002976,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_33.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_33.state')},
'cmd_34': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'137638.53 IOPS 537.65 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'137638.53 IOPS 537.65 MiB/s\n'
'[2022-10-25 23:07:14.742705] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:07:14.742773] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:07:14.742783] bdevperf '
'[2022-10-25 23:07:14.742792] '
'--no-shconf [2022-10-25 '
'23:07:14.742800] -l 0 [2022-10-25 '
'23:07:14.742808] --huge-unlink '
'[2022-10-25 23:07:14.742815] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:07:14.742822] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:07:14.742829] '
'--log-level=user1:6 [2022-10-25 '
'23:07:14.742837] --iova-mode=pa '
'[2022-10-25 23:07:14.742844] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:07:14.742851] '
'--match-allocations [2022-10-25 '
'23:07:14.742858] '
'--file-prefix=spdk_pid83268 '
'[2022-10-25 23:07:14.742866] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:07:14.881095] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:07:14.929550] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:07:14.930229] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_34.output'),
'state': {'begin': 1666724944.4258957,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.125784158706665,
'end': 1666724955.5516798,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_34.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_34.state')},
'cmd_35': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_35.output'),
'state': {'begin': 1666724955.5545845,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7928042411804199,
'end': 1666724956.3473887,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_35.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_35.state')},
'cmd_36': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'122234.74 IOPS 477.48 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'122234.74 IOPS 477.48 MiB/s\n'
'[2022-10-25 23:07:27.731543] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:07:27.731614] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:07:27.731625] bdevperf '
'[2022-10-25 23:07:27.731633] '
'--no-shconf [2022-10-25 '
'23:07:27.731642] -l 0 [2022-10-25 '
'23:07:27.731650] --huge-unlink '
'[2022-10-25 23:07:27.731659] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:07:27.731666] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:07:27.731673] '
'--log-level=user1:6 [2022-10-25 '
'23:07:27.731681] --iova-mode=pa '
'[2022-10-25 23:07:27.731689] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:07:27.731697] '
'--match-allocations [2022-10-25 '
'23:07:27.731705] '
'--file-prefix=spdk_pid83293 '
'[2022-10-25 23:07:27.731713] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:07:27.870333] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:07:27.917497] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:07:27.918172] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_36.output'),
'state': {'begin': 1666724957.4195433,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.131767988204956,
'end': 1666724968.5513113,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_36.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_36.state')},
'cmd_37': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_37.output'),
'state': {'begin': 1666724968.554589,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7912216186523438,
'end': 1666724969.3458107,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_37.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_37.state')},
'cmd_38': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'123370.19 IOPS 481.91 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'123370.19 IOPS 481.91 MiB/s\n'
'[2022-10-25 23:07:40.684960] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:07:40.685028] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:07:40.685039] bdevperf '
'[2022-10-25 23:07:40.685048] '
'--no-shconf [2022-10-25 '
'23:07:40.685057] -l 0 [2022-10-25 '
'23:07:40.685065] --huge-unlink '
'[2022-10-25 23:07:40.685072] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:07:40.685081] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:07:40.685088] '
'--log-level=user1:6 [2022-10-25 '
'23:07:40.685096] --iova-mode=pa '
'[2022-10-25 23:07:40.685103] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:07:40.685112] '
'--match-allocations [2022-10-25 '
'23:07:40.685120] '
'--file-prefix=spdk_pid83318 '
'[2022-10-25 23:07:40.685128] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:07:40.823519] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:07:40.865686] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:07:40.866354] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_38.output'),
'state': {'begin': 1666724970.3716917,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.107799053192139,
'end': 1666724981.4794908,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_38.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_38.state')},
'cmd_39': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_39.output'),
'state': {'begin': 1666724981.4824424,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7807235717773438,
'end': 1666724982.263166,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_39.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_39.state')},
'cmd_40': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'123229.34 IOPS 481.36 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'123229.34 IOPS 481.36 MiB/s\n'
'[2022-10-25 23:07:53.604306] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:07:53.604381] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:07:53.604392] bdevperf '
'[2022-10-25 23:07:53.604401] '
'--no-shconf [2022-10-25 '
'23:07:53.604410] -l 0 [2022-10-25 '
'23:07:53.604418] --huge-unlink '
'[2022-10-25 23:07:53.604427] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:07:53.604434] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:07:53.604442] '
'--log-level=user1:6 [2022-10-25 '
'23:07:53.604450] --iova-mode=pa '
'[2022-10-25 23:07:53.604458] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:07:53.604466] '
'--match-allocations [2022-10-25 '
'23:07:53.604474] '
'--file-prefix=spdk_pid83343 '
'[2022-10-25 23:07:53.604482] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:07:53.742580] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:07:53.813506] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:07:53.814176] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_40.output'),
'state': {'begin': 1666724983.29524,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.133604764938354,
'end': 1666724994.4288447,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_40.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_40.state')},
'cmd_41': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_41.output'),
'state': {'begin': 1666724994.4316053,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7900600433349609,
'end': 1666724995.2216654,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_41.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_41.state')},
'cmd_42': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'123447.21 IOPS 482.22 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'123447.21 IOPS 482.22 MiB/s\n'
'[2022-10-25 23:08:06.563163] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:08:06.563231] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:08:06.563241] bdevperf '
'[2022-10-25 23:08:06.563250] '
'--no-shconf [2022-10-25 '
'23:08:06.563258] -l 0 [2022-10-25 '
'23:08:06.563267] --huge-unlink '
'[2022-10-25 23:08:06.563276] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:08:06.563283] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:08:06.563291] '
'--log-level=user1:6 [2022-10-25 '
'23:08:06.563298] --iova-mode=pa '
'[2022-10-25 23:08:06.563307] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:08:06.563315] '
'--match-allocations [2022-10-25 '
'23:08:06.563323] '
'--file-prefix=spdk_pid83368 '
'[2022-10-25 23:08:06.563331] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:08:06.701480] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:08:06.761538] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:08:06.762219] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_42.output'),
'state': {'begin': 1666724996.2476013,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.12124228477478,
'end': 1666725007.3688436,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_42.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_42.state')},
'cmd_43': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_43.output'),
'state': {'begin': 1666725007.3717415,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7850737571716309,
'end': 1666725008.1568153,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_43.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_43.state')},
'cmd_44': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'132739.00 IOPS 518.51 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'132739.00 IOPS 518.51 MiB/s\n'
'[2022-10-25 23:08:19.508671] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:08:19.508741] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:08:19.508752] bdevperf '
'[2022-10-25 23:08:19.508761] '
'--no-shconf [2022-10-25 '
'23:08:19.508770] -l 0 [2022-10-25 '
'23:08:19.508778] --huge-unlink '
'[2022-10-25 23:08:19.508786] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:08:19.508793] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:08:19.508801] '
'--log-level=user1:6 [2022-10-25 '
'23:08:19.508808] --iova-mode=pa '
'[2022-10-25 23:08:19.508816] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:08:19.508824] '
'--match-allocations [2022-10-25 '
'23:08:19.508832] '
'--file-prefix=spdk_pid83393 '
'[2022-10-25 23:08:19.508840] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:08:19.647435] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:08:19.689565] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:08:19.690233] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_44.output'),
'state': {'begin': 1666725009.1890814,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.122868299484253,
'end': 1666725020.3119497,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_44.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_44.state')},
'cmd_45': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_45.output'),
'state': {'begin': 1666725020.3147836,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8012819290161133,
'end': 1666725021.1160655,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_45.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_45.state')},
'cmd_46': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'132646.77 IOPS 518.15 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'132646.77 IOPS 518.15 MiB/s\n'
'[2022-10-25 23:08:32.474289] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:08:32.474360] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:08:32.474370] bdevperf '
'[2022-10-25 23:08:32.474379] '
'--no-shconf [2022-10-25 '
'23:08:32.474389] -l 0 [2022-10-25 '
'23:08:32.474397] --huge-unlink '
'[2022-10-25 23:08:32.474404] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:08:32.474412] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:08:32.474419] '
'--log-level=user1:6 [2022-10-25 '
'23:08:32.474426] --iova-mode=pa '
'[2022-10-25 23:08:32.474434] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:08:32.474442] '
'--match-allocations [2022-10-25 '
'23:08:32.474451] '
'--file-prefix=spdk_pid83419 '
'[2022-10-25 23:08:32.474458] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:08:32.612935] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:08:32.661606] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:08:32.662286] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_46.output'),
'state': {'begin': 1666725022.1546354,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.117152690887451,
'end': 1666725033.2717881,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_46.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_46.state')},
'cmd_47': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_47.output'),
'state': {'begin': 1666725033.2747984,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7865428924560547,
'end': 1666725034.0613413,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_47.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_47.state')},
'cmd_48': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'132765.93 IOPS 518.62 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'132765.93 IOPS 518.62 MiB/s\n'
'[2022-10-25 23:08:45.526319] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:08:45.526390] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:08:45.526400] bdevperf '
'[2022-10-25 23:08:45.526409] '
'--no-shconf [2022-10-25 '
'23:08:45.526418] -l 0 [2022-10-25 '
'23:08:45.526426] --huge-unlink '
'[2022-10-25 23:08:45.526433] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:08:45.526441] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:08:45.526448] '
'--log-level=user1:6 [2022-10-25 '
'23:08:45.526456] --iova-mode=pa '
'[2022-10-25 23:08:45.526464] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:08:45.526472] '
'--match-allocations [2022-10-25 '
'23:08:45.526479] '
'--file-prefix=spdk_pid83445 '
'[2022-10-25 23:08:45.526487] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:08:45.665426] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:08:45.713532] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:08:45.714200] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_48.output'),
'state': {'begin': 1666725035.0860467,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.245204210281372,
'end': 1666725046.331251,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_48.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_48.state')},
'cmd_49': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_49.output'),
'state': {'begin': 1666725046.3341322,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7895450592041016,
'end': 1666725047.1236773,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_49.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_49.state')},
'cmd_50': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'131267.29 IOPS 512.76 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'131267.29 IOPS 512.76 MiB/s\n'
'[2022-10-25 23:08:58.466029] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:08:58.466097] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:08:58.466108] bdevperf '
'[2022-10-25 23:08:58.466116] '
'--no-shconf [2022-10-25 '
'23:08:58.466125] -l 0 [2022-10-25 '
'23:08:58.466133] --huge-unlink '
'[2022-10-25 23:08:58.466142] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:08:58.466149] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:08:58.466157] '
'--log-level=user1:6 [2022-10-25 '
'23:08:58.466172] --iova-mode=pa '
'[2022-10-25 23:08:58.466180] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:08:58.466189] '
'--match-allocations [2022-10-25 '
'23:08:58.466197] '
'--file-prefix=spdk_pid83471 '
'[2022-10-25 23:08:58.466205] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:08:58.604528] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:08:58.657517] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:08:58.658187] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_50.output'),
'state': {'begin': 1666725048.158771,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.167914152145386,
'end': 1666725059.3266852,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_50.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_50.state')},
'cmd_51': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_51.output'),
'state': {'begin': 1666725059.3299115,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7943465709686279,
'end': 1666725060.124258,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_51.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_51.state')},
'cmd_52': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'131511.69 IOPS 513.72 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'131511.69 IOPS 513.72 MiB/s\n'
'[2022-10-25 23:09:11.464489] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:09:11.464557] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:09:11.464568] bdevperf '
'[2022-10-25 23:09:11.464577] '
'--no-shconf [2022-10-25 '
'23:09:11.464586] -l 0 [2022-10-25 '
'23:09:11.464594] --huge-unlink '
'[2022-10-25 23:09:11.464602] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:09:11.464611] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:09:11.464618] '
'--log-level=user1:6 [2022-10-25 '
'23:09:11.464626] --iova-mode=pa '
'[2022-10-25 23:09:11.464634] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:09:11.464642] '
'--match-allocations [2022-10-25 '
'23:09:11.464650] '
'--file-prefix=spdk_pid83497 '
'[2022-10-25 23:09:11.464658] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:09:11.603992] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:09:11.645517] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:09:11.646198] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_52.output'),
'state': {'begin': 1666725061.1545174,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.164811372756958,
'end': 1666725072.3193288,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_52.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_52.state')},
'cmd_53': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_53.output'),
'state': {'begin': 1666725072.32215,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7881443500518799,
'end': 1666725073.1102943,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_53.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_53.state')},
'cmd_54': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'131606.51 IOPS 514.09 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'131606.51 IOPS 514.09 MiB/s\n'
'[2022-10-25 23:09:24.448368] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:09:24.448436] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:09:24.448447] bdevperf '
'[2022-10-25 23:09:24.448456] '
'--no-shconf [2022-10-25 '
'23:09:24.448465] -l 0 [2022-10-25 '
'23:09:24.448473] --huge-unlink '
'[2022-10-25 23:09:24.448481] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:09:24.448489] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:09:24.448496] '
'--log-level=user1:6 [2022-10-25 '
'23:09:24.448503] --iova-mode=pa '
'[2022-10-25 23:09:24.448511] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:09:24.448520] '
'--match-allocations [2022-10-25 '
'23:09:24.448527] '
'--file-prefix=spdk_pid83522 '
'[2022-10-25 23:09:24.448535] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:09:24.587595] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:09:24.633513] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:09:24.634174] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_54.output'),
'state': {'begin': 1666725074.1374826,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.185035228729248,
'end': 1666725085.3225179,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_54.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_54.state')},
'cmd_55': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_55.output'),
'state': {'begin': 1666725085.3256602,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7825493812561035,
'end': 1666725086.1082096,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_55.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_55.state')},
'cmd_56': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'128907.87 IOPS 503.55 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'128907.87 IOPS 503.55 MiB/s\n'
'[2022-10-25 23:09:37.447829] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:09:37.447899] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:09:37.447909] bdevperf '
'[2022-10-25 23:09:37.447918] '
'--no-shconf [2022-10-25 '
'23:09:37.447926] -l 0 [2022-10-25 '
'23:09:37.447934] --huge-unlink '
'[2022-10-25 23:09:37.447943] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:09:37.447951] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:09:37.447958] '
'--log-level=user1:6 [2022-10-25 '
'23:09:37.447965] --iova-mode=pa '
'[2022-10-25 23:09:37.447973] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:09:37.447981] '
'--match-allocations [2022-10-25 '
'23:09:37.447988] '
'--file-prefix=spdk_pid83547 '
'[2022-10-25 23:09:37.447996] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:09:37.586338] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:09:37.633485] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:09:37.634187] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_56.output'),
'state': {'begin': 1666725087.1419184,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.141113996505737,
'end': 1666725098.2830324,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_56.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_56.state')},
'cmd_57': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_57.output'),
'state': {'begin': 1666725098.2859998,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7840650081634521,
'end': 1666725099.0700648,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_57.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_57.state')},
'cmd_58': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'129471.51 IOPS 505.75 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'129471.51 IOPS 505.75 MiB/s\n'
'[2022-10-25 23:09:50.417714] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:09:50.417786] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:09:50.417797] bdevperf '
'[2022-10-25 23:09:50.417806] '
'--no-shconf [2022-10-25 '
'23:09:50.417815] -l 0 [2022-10-25 '
'23:09:50.417823] --huge-unlink '
'[2022-10-25 23:09:50.417830] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:09:50.417838] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:09:50.417845] '
'--log-level=user1:6 [2022-10-25 '
'23:09:50.417855] --iova-mode=pa '
'[2022-10-25 23:09:50.417864] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:09:50.417872] '
'--match-allocations [2022-10-25 '
'23:09:50.417880] '
'--file-prefix=spdk_pid83574 '
'[2022-10-25 23:09:50.417889] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:09:50.556324] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:09:50.597526] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:09:50.598215] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_58.output'),
'state': {'begin': 1666725100.084945,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.198548555374146,
'end': 1666725111.2834935,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_58.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_58.state')},
'cmd_59': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_59.output'),
'state': {'begin': 1666725111.2860267,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7853360176086426,
'end': 1666725112.0713627,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_59.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_59.state')},
'cmd_60': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'134840.52 IOPS 526.72 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'134840.52 IOPS 526.72 MiB/s\n'
'[2022-10-25 23:10:03.412061] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:10:03.412135] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:10:03.412146] bdevperf '
'[2022-10-25 23:10:03.412155] '
'--no-shconf [2022-10-25 '
'23:10:03.412164] -l 0 [2022-10-25 '
'23:10:03.412171] --huge-unlink '
'[2022-10-25 23:10:03.412179] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:10:03.412187] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:10:03.412195] '
'--log-level=user1:6 [2022-10-25 '
'23:10:03.412204] --iova-mode=pa '
'[2022-10-25 23:10:03.412212] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:10:03.412220] '
'--match-allocations [2022-10-25 '
'23:10:03.412227] '
'--file-prefix=spdk_pid83599 '
'[2022-10-25 23:10:03.412236] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:10:03.550916] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:10:03.597661] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:10:03.598365] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_60.output'),
'state': {'begin': 1666725113.1110542,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 2 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.184890270233154,
'end': 1666725124.2959445,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_60.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_60.state')},
'cmd_61': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_61.output'),
'state': {'begin': 1666725124.2987037,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7947700023651123,
'end': 1666725125.0934737,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_61.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_61.state')},
'cmd_62': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'220666.80 IOPS 861.98 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'220666.80 IOPS 861.98 MiB/s\n'
'[2022-10-25 23:10:16.626846] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:10:16.626916] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:10:16.626927] bdevperf '
'[2022-10-25 23:10:16.626935] '
'--no-shconf [2022-10-25 '
'23:10:16.626943] -l 0 [2022-10-25 '
'23:10:16.626951] --huge-unlink '
'[2022-10-25 23:10:16.626958] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:10:16.626966] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:10:16.626973] '
'--log-level=user1:6 [2022-10-25 '
'23:10:16.626980] --iova-mode=pa '
'[2022-10-25 23:10:16.626987] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:10:16.626995] '
'--match-allocations [2022-10-25 '
'23:10:16.627002] '
'--file-prefix=spdk_pid83624 '
'[2022-10-25 23:10:16.627009] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:10:16.765360] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:10:16.817512] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:10:16.818222] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_62.output'),
'state': {'begin': 1666725126.2161548,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.227761268615723,
'end': 1666725137.443916,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_62.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_62.state')},
'cmd_63': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_63.output'),
'state': {'begin': 1666725137.446754,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7797489166259766,
'end': 1666725138.226503,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_63.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_63.state')},
'cmd_64': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'216380.02 IOPS 845.23 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'216380.02 IOPS 845.23 MiB/s\n'
'[2022-10-25 23:10:29.570713] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:10:29.570782] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:10:29.570792] bdevperf '
'[2022-10-25 23:10:29.570801] '
'--no-shconf [2022-10-25 '
'23:10:29.570810] -l 0 [2022-10-25 '
'23:10:29.570818] --huge-unlink '
'[2022-10-25 23:10:29.570826] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:10:29.570834] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:10:29.570842] '
'--log-level=user1:6 [2022-10-25 '
'23:10:29.570850] --iova-mode=pa '
'[2022-10-25 23:10:29.570859] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:10:29.570867] '
'--match-allocations [2022-10-25 '
'23:10:29.570876] '
'--file-prefix=spdk_pid83649 '
'[2022-10-25 23:10:29.570884] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:10:29.709427] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:10:29.753600] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:10:29.754303] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_64.output'),
'state': {'begin': 1666725139.2636514,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.116545677185059,
'end': 1666725150.380197,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_64.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_64.state')},
'cmd_65': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_65.output'),
'state': {'begin': 1666725150.3830373,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7887518405914307,
'end': 1666725151.1717892,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_65.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_65.state')},
'cmd_66': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'216607.90 IOPS 846.12 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'216607.90 IOPS 846.12 MiB/s\n'
'[2022-10-25 23:10:42.527926] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:10:42.527995] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:10:42.528006] bdevperf '
'[2022-10-25 23:10:42.528015] '
'--no-shconf [2022-10-25 '
'23:10:42.528023] -l 0 [2022-10-25 '
'23:10:42.528030] --huge-unlink '
'[2022-10-25 23:10:42.528039] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:10:42.528047] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:10:42.528054] '
'--log-level=user1:6 [2022-10-25 '
'23:10:42.528080] --iova-mode=pa '
'[2022-10-25 23:10:42.528088] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:10:42.528096] '
'--match-allocations [2022-10-25 '
'23:10:42.528104] '
'--file-prefix=spdk_pid83674 '
'[2022-10-25 23:10:42.528119] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:10:42.666440] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:10:42.709521] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:10:42.710222] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_66.output'),
'state': {'begin': 1666725152.213005,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.126810789108276,
'end': 1666725163.3398159,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_66.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_66.state')},
'cmd_67': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_67.output'),
'state': {'begin': 1666725163.3429255,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.789379358291626,
'end': 1666725164.132305,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_67.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_67.state')},
'cmd_68': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'209760.64 IOPS 819.38 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'209760.64 IOPS 819.38 MiB/s\n'
'[2022-10-25 23:10:55.470262] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:10:55.470332] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:10:55.470343] bdevperf '
'[2022-10-25 23:10:55.470352] '
'--no-shconf [2022-10-25 '
'23:10:55.470361] -l 0 [2022-10-25 '
'23:10:55.470369] --huge-unlink '
'[2022-10-25 23:10:55.470378] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:10:55.470386] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:10:55.470393] '
'--log-level=user1:6 [2022-10-25 '
'23:10:55.470402] --iova-mode=pa '
'[2022-10-25 23:10:55.470409] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:10:55.470417] '
'--match-allocations [2022-10-25 '
'23:10:55.470425] '
'--file-prefix=spdk_pid83699 '
'[2022-10-25 23:10:55.470433] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:10:55.609173] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:10:55.669534] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:10:55.670217] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_68.output'),
'state': {'begin': 1666725165.1595466,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.120136499404907,
'end': 1666725176.279683,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_68.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_68.state')},
'cmd_69': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_69.output'),
'state': {'begin': 1666725176.28263,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7875056266784668,
'end': 1666725177.0701356,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_69.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_69.state')},
'cmd_70': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'209376.05 IOPS 817.88 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'209376.05 IOPS 817.88 MiB/s\n'
'[2022-10-25 23:11:08.404264] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:11:08.404334] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:11:08.404345] bdevperf '
'[2022-10-25 23:11:08.404354] '
'--no-shconf [2022-10-25 '
'23:11:08.404362] -l 0 [2022-10-25 '
'23:11:08.404370] --huge-unlink '
'[2022-10-25 23:11:08.404379] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:11:08.404386] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:11:08.404394] '
'--log-level=user1:6 [2022-10-25 '
'23:11:08.404402] --iova-mode=pa '
'[2022-10-25 23:11:08.404410] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:11:08.404418] '
'--match-allocations [2022-10-25 '
'23:11:08.404426] '
'--file-prefix=spdk_pid83724 '
'[2022-10-25 23:11:08.404434] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:11:08.542984] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:11:08.589571] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:11:08.590268] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_70.output'),
'state': {'begin': 1666725178.096931,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.102919816970825,
'end': 1666725189.1998508,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_70.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_70.state')},
'cmd_71': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_71.output'),
'state': {'begin': 1666725189.2025259,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8047060966491699,
'end': 1666725190.007232,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_71.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_71.state')},
'cmd_72': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'209107.46 IOPS 816.83 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'209107.46 IOPS 816.83 MiB/s\n'
'[2022-10-25 23:11:21.347093] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:11:21.347171] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:11:21.347182] bdevperf '
'[2022-10-25 23:11:21.347191] '
'--no-shconf [2022-10-25 '
'23:11:21.347200] -l 0 [2022-10-25 '
'23:11:21.347208] --huge-unlink '
'[2022-10-25 23:11:21.347217] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:11:21.347225] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:11:21.347234] '
'--log-level=user1:6 [2022-10-25 '
'23:11:21.347242] --iova-mode=pa '
'[2022-10-25 23:11:21.347250] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:11:21.347258] '
'--match-allocations [2022-10-25 '
'23:11:21.347266] '
'--file-prefix=spdk_pid83749 '
'[2022-10-25 23:11:21.347274] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:11:21.485620] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:11:21.541528] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:11:21.542206] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_72.output'),
'state': {'begin': 1666725191.0355625,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.117104768753052,
'end': 1666725202.1526673,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_72.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_72.state')},
'cmd_73': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_73.output'),
'state': {'begin': 1666725202.1556916,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7884480953216553,
'end': 1666725202.9441397,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_73.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_73.state')},
'cmd_74': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'221094.82 IOPS 863.65 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'221094.82 IOPS 863.65 MiB/s\n'
'[2022-10-25 23:11:34.290214] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:11:34.290284] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:11:34.290294] bdevperf '
'[2022-10-25 23:11:34.290303] '
'--no-shconf [2022-10-25 '
'23:11:34.290311] -l 0 [2022-10-25 '
'23:11:34.290319] --huge-unlink '
'[2022-10-25 23:11:34.290326] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:11:34.290335] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:11:34.290342] '
'--log-level=user1:6 [2022-10-25 '
'23:11:34.290350] --iova-mode=pa '
'[2022-10-25 23:11:34.290376] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:11:34.290385] '
'--match-allocations [2022-10-25 '
'23:11:34.290393] '
'--file-prefix=spdk_pid83774 '
'[2022-10-25 23:11:34.290409] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:11:34.428902] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:11:34.469602] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:11:34.470285] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_74.output'),
'state': {'begin': 1666725203.9746785,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.109240770339966,
'end': 1666725215.0839193,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_74.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_74.state')},
'cmd_75': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_75.output'),
'state': {'begin': 1666725215.0867696,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7892894744873047,
'end': 1666725215.876059,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_75.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_75.state')},
'cmd_76': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'221310.00 IOPS 864.49 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'221310.00 IOPS 864.49 MiB/s\n'
'[2022-10-25 23:11:47.217369] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:11:47.217440] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:11:47.217451] bdevperf '
'[2022-10-25 23:11:47.217461] '
'--no-shconf [2022-10-25 '
'23:11:47.217468] -l 0 [2022-10-25 '
'23:11:47.217476] --huge-unlink '
'[2022-10-25 23:11:47.217484] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:11:47.217491] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:11:47.217498] '
'--log-level=user1:6 [2022-10-25 '
'23:11:47.217505] --iova-mode=pa '
'[2022-10-25 23:11:47.217513] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:11:47.217520] '
'--match-allocations [2022-10-25 '
'23:11:47.217527] '
'--file-prefix=spdk_pid83799 '
'[2022-10-25 23:11:47.217534] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:11:47.355898] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:11:47.397531] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:11:47.398212] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_76.output'),
'state': {'begin': 1666725216.9129891,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.102845430374146,
'end': 1666725228.0158346,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_76.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_76.state')},
'cmd_77': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_77.output'),
'state': {'begin': 1666725228.0186582,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7831084728240967,
'end': 1666725228.8017666,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_77.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_77.state')},
'cmd_78': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'221270.62 IOPS 864.34 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'221270.62 IOPS 864.34 MiB/s\n'
'[2022-10-25 23:12:00.149272] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:12:00.149341] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:12:00.149351] bdevperf '
'[2022-10-25 23:12:00.149360] '
'--no-shconf [2022-10-25 '
'23:12:00.149369] -l 0 [2022-10-25 '
'23:12:00.149377] --huge-unlink '
'[2022-10-25 23:12:00.149386] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:12:00.149393] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:12:00.149401] '
'--log-level=user1:6 [2022-10-25 '
'23:12:00.149409] --iova-mode=pa '
'[2022-10-25 23:12:00.149418] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:12:00.149426] '
'--match-allocations [2022-10-25 '
'23:12:00.149434] '
'--file-prefix=spdk_pid83825 '
'[2022-10-25 23:12:00.149442] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:12:00.287764] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:12:00.329617] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:12:00.330281] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_78.output'),
'state': {'begin': 1666725229.8440409,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.098433256149292,
'end': 1666725240.9424741,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_78.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_78.state')},
'cmd_79': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_79.output'),
'state': {'begin': 1666725240.9454057,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7916970252990723,
'end': 1666725241.7371027,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_79.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_79.state')},
'cmd_80': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'219908.59 IOPS 859.02 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'219908.59 IOPS 859.02 MiB/s\n'
'[2022-10-25 23:12:13.069798] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:12:13.069876] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:12:13.069887] bdevperf '
'[2022-10-25 23:12:13.069896] '
'--no-shconf [2022-10-25 '
'23:12:13.069905] -l 0 [2022-10-25 '
'23:12:13.069912] --huge-unlink '
'[2022-10-25 23:12:13.069921] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:12:13.069929] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:12:13.069936] '
'--log-level=user1:6 [2022-10-25 '
'23:12:13.069945] --iova-mode=pa '
'[2022-10-25 23:12:13.069953] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:12:13.069961] '
'--match-allocations [2022-10-25 '
'23:12:13.069969] '
'--file-prefix=spdk_pid83850 '
'[2022-10-25 23:12:13.069977] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:12:13.208316] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:12:13.249582] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:12:13.250261] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_80.output'),
'state': {'begin': 1666725242.7690065,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.156771421432495,
'end': 1666725253.925778,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_80.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_80.state')},
'cmd_81': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_81.output'),
'state': {'begin': 1666725253.9289076,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7852492332458496,
'end': 1666725254.7141569,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_81.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_81.state')},
'cmd_82': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'220284.56 IOPS 860.49 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'220284.56 IOPS 860.49 MiB/s\n'
'[2022-10-25 23:12:26.055510] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:12:26.055579] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:12:26.055589] bdevperf '
'[2022-10-25 23:12:26.055598] '
'--no-shconf [2022-10-25 '
'23:12:26.055606] -l 0 [2022-10-25 '
'23:12:26.055614] --huge-unlink '
'[2022-10-25 23:12:26.055621] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:12:26.055628] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:12:26.055636] '
'--log-level=user1:6 [2022-10-25 '
'23:12:26.055643] --iova-mode=pa '
'[2022-10-25 23:12:26.055650] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:12:26.055657] '
'--match-allocations [2022-10-25 '
'23:12:26.055665] '
'--file-prefix=spdk_pid83875 '
'[2022-10-25 23:12:26.055672] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:12:26.194021] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:12:26.245587] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:12:26.246263] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_82.output'),
'state': {'begin': 1666725255.74967,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.196541547775269,
'end': 1666725266.9462116,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_82.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_82.state')},
'cmd_83': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_83.output'),
'state': {'begin': 1666725266.949083,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7843360900878906,
'end': 1666725267.7334192,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_83.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_83.state')},
'cmd_84': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'220435.91 IOPS 861.08 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'220435.91 IOPS 861.08 MiB/s\n'
'[2022-10-25 23:12:39.062862] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:12:39.062933] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:12:39.062944] bdevperf '
'[2022-10-25 23:12:39.062952] '
'--no-shconf [2022-10-25 '
'23:12:39.062961] -l 0 [2022-10-25 '
'23:12:39.062969] --huge-unlink '
'[2022-10-25 23:12:39.062978] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:12:39.062985] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:12:39.062992] '
'--log-level=user1:6 [2022-10-25 '
'23:12:39.063000] --iova-mode=pa '
'[2022-10-25 23:12:39.063008] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:12:39.063016] '
'--match-allocations [2022-10-25 '
'23:12:39.063024] '
'--file-prefix=spdk_pid83901 '
'[2022-10-25 23:12:39.063033] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:12:39.201528] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:12:39.258472] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:12:39.259141] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_84.output'),
'state': {'begin': 1666725268.7572954,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.184575080871582,
'end': 1666725279.9418705,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_84.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_84.state')},
'cmd_85': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_85.output'),
'state': {'begin': 1666725279.944596,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7948479652404785,
'end': 1666725280.739444,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_85.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_85.state')},
'cmd_86': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'218110.54 IOPS 851.99 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'218110.54 IOPS 851.99 MiB/s\n'
'[2022-10-25 23:12:52.069012] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:12:52.069081] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:12:52.069091] bdevperf '
'[2022-10-25 23:12:52.069100] '
'--no-shconf [2022-10-25 '
'23:12:52.069108] -l 0 [2022-10-25 '
'23:12:52.069116] --huge-unlink '
'[2022-10-25 23:12:52.069124] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:12:52.069131] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:12:52.069139] '
'--log-level=user1:6 [2022-10-25 '
'23:12:52.069146] --iova-mode=pa '
'[2022-10-25 23:12:52.069153] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:12:52.069161] '
'--match-allocations [2022-10-25 '
'23:12:52.069168] '
'--file-prefix=spdk_pid83926 '
'[2022-10-25 23:12:52.069176] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:12:52.207851] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:12:52.253574] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:12:52.254257] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_86.output'),
'state': {'begin': 1666725281.7638643,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.18337607383728,
'end': 1666725292.9472404,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_86.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_86.state')},
'cmd_87': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_87.output'),
'state': {'begin': 1666725292.9499643,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7913548946380615,
'end': 1666725293.7413192,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_87.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_87.state')},
'cmd_88': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'217774.07 IOPS 850.68 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'217774.07 IOPS 850.68 MiB/s\n'
'[2022-10-25 23:13:05.092769] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:13:05.092840] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:13:05.092850] bdevperf '
'[2022-10-25 23:13:05.092859] '
'--no-shconf [2022-10-25 '
'23:13:05.092868] -l 0 [2022-10-25 '
'23:13:05.092876] --huge-unlink '
'[2022-10-25 23:13:05.092885] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:13:05.092894] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:13:05.092901] '
'--log-level=user1:6 [2022-10-25 '
'23:13:05.092909] --iova-mode=pa '
'[2022-10-25 23:13:05.092918] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:13:05.092926] '
'--match-allocations [2022-10-25 '
'23:13:05.092934] '
'--file-prefix=spdk_pid83952 '
'[2022-10-25 23:13:05.092943] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:13:05.231410] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:13:05.285594] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:13:05.286255] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_88.output'),
'state': {'begin': 1666725294.7792244,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.191197872161865,
'end': 1666725305.9704223,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_88.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_88.state')},
'cmd_89': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_89.output'),
'state': {'begin': 1666725305.9733083,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8015813827514648,
'end': 1666725306.7748897,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_89.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_89.state')},
'cmd_90': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'217800.45 IOPS 850.78 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'217800.45 IOPS 850.78 MiB/s\n'
'[2022-10-25 23:13:18.110405] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:13:18.110473] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:13:18.110484] bdevperf '
'[2022-10-25 23:13:18.110492] '
'--no-shconf [2022-10-25 '
'23:13:18.110500] -l 0 [2022-10-25 '
'23:13:18.110508] --huge-unlink '
'[2022-10-25 23:13:18.110515] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:13:18.110523] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:13:18.110530] '
'--log-level=user1:6 [2022-10-25 '
'23:13:18.110538] --iova-mode=pa '
'[2022-10-25 23:13:18.110544] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:13:18.110552] '
'--match-allocations [2022-10-25 '
'23:13:18.110559] '
'--file-prefix=spdk_pid83977 '
'[2022-10-25 23:13:18.110566] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:13:18.249238] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:13:18.293492] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:13:18.294175] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_90.output'),
'state': {'begin': 1666725307.8022592,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 4 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.190491676330566,
'end': 1666725318.992751,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_90.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_90.state')},
'cmd_91': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_91.output'),
'state': {'begin': 1666725319.0001276,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7905046939849854,
'end': 1666725319.7906322,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_91.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_91.state')},
'cmd_92': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'0.06 IOPS 0.00 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total '
': 0.06 IOPS 0.00 '
'MiB/s\n'
'[2022-10-25 23:13:31.240110] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:13:31.240182] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:13:31.240193] bdevperf '
'[2022-10-25 23:13:31.240202] '
'--no-shconf [2022-10-25 '
'23:13:31.240210] -l 0 [2022-10-25 '
'23:13:31.240219] --huge-unlink '
'[2022-10-25 23:13:31.240227] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:13:31.240234] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:13:31.240241] '
'--log-level=user1:6 [2022-10-25 '
'23:13:31.240249] --iova-mode=pa '
'[2022-10-25 23:13:31.240257] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:13:31.240265] '
'--match-allocations [2022-10-25 '
'23:13:31.240272] '
'--file-prefix=spdk_pid84002 '
'[2022-10-25 23:13:31.240279] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:13:31.378840] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:13:31.433486] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:13:31.434166] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_92.output'),
'state': {'begin': 1666725320.8388972,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 398.423148393631,
'end': 1666725719.2620456,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_92.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_92.state')},
'cmd_93': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_93.output'),
'state': {'begin': 1666725719.2653906,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8903448581695557,
'end': 1666725720.1557355,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_93.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_93.state')},
'cmd_94': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'222460.54 IOPS 868.99 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'222460.54 IOPS 868.99 MiB/s\n'
'[2022-10-25 23:20:11.844287] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:20:11.844351] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:20:11.844362] bdevperf '
'[2022-10-25 23:20:11.844371] '
'--no-shconf [2022-10-25 '
'23:20:11.844380] -l 0 [2022-10-25 '
'23:20:11.844388] --huge-unlink '
'[2022-10-25 23:20:11.844396] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:20:11.844405] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:20:11.844412] '
'--log-level=user1:6 [2022-10-25 '
'23:20:11.844438] --iova-mode=pa '
'[2022-10-25 23:20:11.844448] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:20:11.844456] '
'--match-allocations [2022-10-25 '
'23:20:11.844465] '
'--file-prefix=spdk_pid84034 '
'[2022-10-25 23:20:11.844481] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:20:11.982803] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:20:12.037539] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:20:12.038213] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_94.output'),
'state': {'begin': 1666725721.347637,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.330201387405396,
'end': 1666725732.6778383,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_94.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_94.state')},
'cmd_95': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_95.output'),
'state': {'begin': 1666725732.6809075,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.786815881729126,
'end': 1666725733.4677234,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_95.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_95.state')},
'cmd_96': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'222470.91 IOPS 869.03 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'222470.91 IOPS 869.03 MiB/s\n'
'[2022-10-25 23:20:24.779813] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:20:24.779881] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:20:24.779892] bdevperf '
'[2022-10-25 23:20:24.779901] '
'--no-shconf [2022-10-25 '
'23:20:24.779910] -l 0 [2022-10-25 '
'23:20:24.779918] --huge-unlink '
'[2022-10-25 23:20:24.779927] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:20:24.779934] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:20:24.779942] '
'--log-level=user1:6 [2022-10-25 '
'23:20:24.779950] --iova-mode=pa '
'[2022-10-25 23:20:24.779958] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:20:24.779966] '
'--match-allocations [2022-10-25 '
'23:20:24.779974] '
'--file-prefix=spdk_pid84061 '
'[2022-10-25 23:20:24.779983] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:20:24.918965] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:20:24.961539] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:20:24.962206] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_96.output'),
'state': {'begin': 1666725734.491195,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.11571478843689,
'end': 1666725745.6069098,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_96.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_96.state')},
'cmd_97': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_97.output'),
'state': {'begin': 1666725745.6098454,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7851316928863525,
'end': 1666725746.394977,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_97.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_97.state')},
'cmd_98': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'223754.18 IOPS 874.04 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'223754.18 IOPS 874.04 MiB/s\n'
'[2022-10-25 23:20:37.720662] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-25 23:20:37.720732] [ '
'DPDK EAL parameters: [2022-10-25 '
'23:20:37.720742] bdevperf '
'[2022-10-25 23:20:37.720751] '
'--no-shconf [2022-10-25 '
'23:20:37.720760] -l 0 [2022-10-25 '
'23:20:37.720768] --huge-unlink '
'[2022-10-25 23:20:37.720776] '
'--log-level=lib.eal:6 [2022-10-25 '
'23:20:37.720784] '
'--log-level=lib.cryptodev:5 '
'[2022-10-25 23:20:37.720792] '
'--log-level=user1:6 [2022-10-25 '
'23:20:37.720800] --iova-mode=pa '
'[2022-10-25 23:20:37.720809] '
'--base-virtaddr=0x200000000000 '
'[2022-10-25 23:20:37.720817] '
'--match-allocations [2022-10-25 '
'23:20:37.720825] '
'--file-prefix=spdk_pid84086 '
'[2022-10-25 23:20:37.720833] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-25 23:20:37.859472] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-25 23:20:37.909520] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-25 23:20:37.910195] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_98.output'),
'state': {'begin': 1666725747.426107,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 8 -o 4096 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.111924409866333,
'end': 1666725758.5380313,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_98.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_98.state')},
'cmd_99': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_99.output'),
'state': {'begin': 1666725758.540498,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7880303859710693,
'end': 1666725759.3285284,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_99.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_99.state')}}},
'id': '001_bdevperf',
'name': 'bdevperf',
'nr': 1,
'status': {'elapsed': 1166.063898563385,
'failed': 0,
'passed': 1,
'skipped': 0,
'started': 0.0},
'uses': 'bdevperf',
'with': {'iodepths': [1, 2, 4, 8], 'iosizes': ['4K'], 'repetitions': 3}}
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 96312.06 IOPS 376.22 MiB/s
=============================================================
Total : 96312.06 IOPS 376.22 MiB/s
[2022-10-25 23:03:47.429048] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:03:47.429120] [ DPDK EAL parameters: [2022-10-25 23:03:47.429132] bdevperf [2022-10-25 23:03:47.429142] --no-shconf [2022-10-25 23:03:47.429151] -l 0 [2022-10-25 23:03:47.429169] --huge-unlink [2022-10-25 23:03:47.429179] --log-level=lib.eal:6 [2022-10-25 23:03:47.429188] --log-level=lib.cryptodev:5 [2022-10-25 23:03:47.429196] --log-level=user1:6 [2022-10-25 23:03:47.429205] --iova-mode=pa [2022-10-25 23:03:47.429219] --base-virtaddr=0x200000000000 [2022-10-25 23:03:47.429227] --match-allocations [2022-10-25 23:03:47.429235] --file-prefix=spdk_pid82863 [2022-10-25 23:03:47.429243] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:03:47.567686] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:03:47.609539] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:03:47.610229] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 91332.77 IOPS 356.77 MiB/s
=============================================================
Total : 91332.77 IOPS 356.77 MiB/s
[2022-10-25 23:04:00.370178] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:04:00.370248] [ DPDK EAL parameters: [2022-10-25 23:04:00.370259] bdevperf [2022-10-25 23:04:00.370268] --no-shconf [2022-10-25 23:04:00.370277] -l 0 [2022-10-25 23:04:00.370285] --huge-unlink [2022-10-25 23:04:00.370293] --log-level=lib.eal:6 [2022-10-25 23:04:00.370302] --log-level=lib.cryptodev:5 [2022-10-25 23:04:00.370309] --log-level=user1:6 [2022-10-25 23:04:00.370317] --iova-mode=pa [2022-10-25 23:04:00.370325] --base-virtaddr=0x200000000000 [2022-10-25 23:04:00.370333] --match-allocations [2022-10-25 23:04:00.370341] --file-prefix=spdk_pid82888 [2022-10-25 23:04:00.370350] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:04:00.508790] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:04:00.557574] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:04:00.558236] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 91051.09 IOPS 355.67 MiB/s
=============================================================
Total : 91051.09 IOPS 355.67 MiB/s
[2022-10-25 23:04:13.314791] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:04:13.314863] [ DPDK EAL parameters: [2022-10-25 23:04:13.314874] bdevperf [2022-10-25 23:04:13.314882] --no-shconf [2022-10-25 23:04:13.314890] -l 0 [2022-10-25 23:04:13.314898] --huge-unlink [2022-10-25 23:04:13.314905] --log-level=lib.eal:6 [2022-10-25 23:04:13.314913] --log-level=lib.cryptodev:5 [2022-10-25 23:04:13.314920] --log-level=user1:6 [2022-10-25 23:04:13.314927] --iova-mode=pa [2022-10-25 23:04:13.314935] --base-virtaddr=0x200000000000 [2022-10-25 23:04:13.314942] --match-allocations [2022-10-25 23:04:13.314949] --file-prefix=spdk_pid82913 [2022-10-25 23:04:13.314956] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:04:13.453309] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:04:13.493561] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:04:13.494240] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
rm: cannot remove '/tmp/spdk_bdev_uring_foo.conf': No such file or directory
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 87268.14 IOPS 340.89 MiB/s
=============================================================
Total : 87268.14 IOPS 340.89 MiB/s
[2022-10-25 23:04:26.232081] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:04:26.232151] [ DPDK EAL parameters: [2022-10-25 23:04:26.232162] bdevperf [2022-10-25 23:04:26.232171] --no-shconf [2022-10-25 23:04:26.232180] -l 0 [2022-10-25 23:04:26.232189] --huge-unlink [2022-10-25 23:04:26.232197] --log-level=lib.eal:6 [2022-10-25 23:04:26.232205] --log-level=lib.cryptodev:5 [2022-10-25 23:04:26.232212] --log-level=user1:6 [2022-10-25 23:04:26.232221] --iova-mode=pa [2022-10-25 23:04:26.232229] --base-virtaddr=0x200000000000 [2022-10-25 23:04:26.232237] --match-allocations [2022-10-25 23:04:26.232245] --file-prefix=spdk_pid82940 [2022-10-25 23:04:26.232253] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:04:26.370496] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:04:26.425602] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:04:26.426289] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 87338.70 IOPS 341.17 MiB/s
=============================================================
Total : 87338.70 IOPS 341.17 MiB/s
[2022-10-25 23:04:39.195818] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:04:39.195889] [ DPDK EAL parameters: [2022-10-25 23:04:39.195900] bdevperf [2022-10-25 23:04:39.195909] --no-shconf [2022-10-25 23:04:39.195918] -l 0 [2022-10-25 23:04:39.195926] --huge-unlink [2022-10-25 23:04:39.195934] --log-level=lib.eal:6 [2022-10-25 23:04:39.195942] --log-level=lib.cryptodev:5 [2022-10-25 23:04:39.195949] --log-level=user1:6 [2022-10-25 23:04:39.195957] --iova-mode=pa [2022-10-25 23:04:39.195966] --base-virtaddr=0x200000000000 [2022-10-25 23:04:39.195973] --match-allocations [2022-10-25 23:04:39.195981] --file-prefix=spdk_pid82965 [2022-10-25 23:04:39.195989] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:04:39.333951] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:04:39.381531] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:04:39.382192] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 223720.83 IOPS 873.91 MiB/s
=============================================================
Total : 223720.83 IOPS 873.91 MiB/s
[2022-10-25 23:20:50.653460] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:20:50.653531] [ DPDK EAL parameters: [2022-10-25 23:20:50.653542] bdevperf [2022-10-25 23:20:50.653551] --no-shconf [2022-10-25 23:20:50.653560] -l 0 [2022-10-25 23:20:50.653567] --huge-unlink [2022-10-25 23:20:50.653575] --log-level=lib.eal:6 [2022-10-25 23:20:50.653583] --log-level=lib.cryptodev:5 [2022-10-25 23:20:50.653591] --log-level=user1:6 [2022-10-25 23:20:50.653599] --iova-mode=pa [2022-10-25 23:20:50.653607] --base-virtaddr=0x200000000000 [2022-10-25 23:20:50.653614] --match-allocations [2022-10-25 23:20:50.653622] --file-prefix=spdk_pid84111 [2022-10-25 23:20:50.653629] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:20:50.792163] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:20:50.833572] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:20:50.834238] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 223708.71 IOPS 873.86 MiB/s
=============================================================
Total : 223708.71 IOPS 873.86 MiB/s
[2022-10-25 23:21:03.647189] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:21:03.647258] [ DPDK EAL parameters: [2022-10-25 23:21:03.647268] bdevperf [2022-10-25 23:21:03.647277] --no-shconf [2022-10-25 23:21:03.647286] -l 0 [2022-10-25 23:21:03.647295] --huge-unlink [2022-10-25 23:21:03.647303] --log-level=lib.eal:6 [2022-10-25 23:21:03.647311] --log-level=lib.cryptodev:5 [2022-10-25 23:21:03.647318] --log-level=user1:6 [2022-10-25 23:21:03.647326] --iova-mode=pa [2022-10-25 23:21:03.647334] --base-virtaddr=0x200000000000 [2022-10-25 23:21:03.647342] --match-allocations [2022-10-25 23:21:03.647350] --file-prefix=spdk_pid84137 [2022-10-25 23:21:03.647359] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:21:03.785565] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:21:03.829482] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:21:03.830158] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 223714.32 IOPS 873.88 MiB/s
=============================================================
Total : 223714.32 IOPS 873.88 MiB/s
[2022-10-25 23:21:16.578379] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:21:16.578446] [ DPDK EAL parameters: [2022-10-25 23:21:16.578457] bdevperf [2022-10-25 23:21:16.578465] --no-shconf [2022-10-25 23:21:16.578474] -l 0 [2022-10-25 23:21:16.578482] --huge-unlink [2022-10-25 23:21:16.578489] --log-level=lib.eal:6 [2022-10-25 23:21:16.578497] --log-level=lib.cryptodev:5 [2022-10-25 23:21:16.578504] --log-level=user1:6 [2022-10-25 23:21:16.578512] --iova-mode=pa [2022-10-25 23:21:16.578519] --base-virtaddr=0x200000000000 [2022-10-25 23:21:16.578526] --match-allocations [2022-10-25 23:21:16.578534] --file-prefix=spdk_pid84162 [2022-10-25 23:21:16.578541] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:21:16.717093] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:21:16.765505] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:21:16.766176] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 223656.87 IOPS 873.66 MiB/s
=============================================================
Total : 223656.87 IOPS 873.66 MiB/s
[2022-10-25 23:21:29.521276] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:21:29.521350] [ DPDK EAL parameters: [2022-10-25 23:21:29.521362] bdevperf [2022-10-25 23:21:29.521371] --no-shconf [2022-10-25 23:21:29.521380] -l 0 [2022-10-25 23:21:29.521388] --huge-unlink [2022-10-25 23:21:29.521396] --log-level=lib.eal:6 [2022-10-25 23:21:29.521404] --log-level=lib.cryptodev:5 [2022-10-25 23:21:29.521412] --log-level=user1:6 [2022-10-25 23:21:29.521420] --iova-mode=pa [2022-10-25 23:21:29.521428] --base-virtaddr=0x200000000000 [2022-10-25 23:21:29.521436] --match-allocations [2022-10-25 23:21:29.521443] --file-prefix=spdk_pid84187 [2022-10-25 23:21:29.521454] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:21:29.660283] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:21:29.701545] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:21:29.702222] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 223746.27 IOPS 874.01 MiB/s
=============================================================
Total : 223746.27 IOPS 874.01 MiB/s
[2022-10-25 23:21:42.426968] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:21:42.427037] [ DPDK EAL parameters: [2022-10-25 23:21:42.427047] bdevperf [2022-10-25 23:21:42.427056] --no-shconf [2022-10-25 23:21:42.427063] -l 0 [2022-10-25 23:21:42.427071] --huge-unlink [2022-10-25 23:21:42.427078] --log-level=lib.eal:6 [2022-10-25 23:21:42.427086] --log-level=lib.cryptodev:5 [2022-10-25 23:21:42.427093] --log-level=user1:6 [2022-10-25 23:21:42.427100] --iova-mode=pa [2022-10-25 23:21:42.427108] --base-virtaddr=0x200000000000 [2022-10-25 23:21:42.427125] --match-allocations [2022-10-25 23:21:42.427132] --file-prefix=spdk_pid84213 [2022-10-25 23:21:42.427139] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:21:42.565813] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:21:42.613477] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:21:42.614155] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 223741.17 IOPS 873.99 MiB/s
=============================================================
Total : 223741.17 IOPS 873.99 MiB/s
[2022-10-25 23:21:55.355943] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:21:55.356022] [ DPDK EAL parameters: [2022-10-25 23:21:55.356033] bdevperf [2022-10-25 23:21:55.356042] --no-shconf [2022-10-25 23:21:55.356050] -l 0 [2022-10-25 23:21:55.356057] --huge-unlink [2022-10-25 23:21:55.356065] --log-level=lib.eal:6 [2022-10-25 23:21:55.356072] --log-level=lib.cryptodev:5 [2022-10-25 23:21:55.356079] --log-level=user1:6 [2022-10-25 23:21:55.356087] --iova-mode=pa [2022-10-25 23:21:55.356094] --base-virtaddr=0x200000000000 [2022-10-25 23:21:55.356102] --match-allocations [2022-10-25 23:21:55.356109] --file-prefix=spdk_pid84238 [2022-10-25 23:21:55.356116] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:21:55.494518] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:21:55.549627] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:21:55.550308] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 223785.38 IOPS 874.16 MiB/s
=============================================================
Total : 223785.38 IOPS 874.16 MiB/s
[2022-10-25 23:22:08.359371] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:22:08.359442] [ DPDK EAL parameters: [2022-10-25 23:22:08.359452] bdevperf [2022-10-25 23:22:08.359461] --no-shconf [2022-10-25 23:22:08.359470] -l 0 [2022-10-25 23:22:08.359478] --huge-unlink [2022-10-25 23:22:08.359486] --log-level=lib.eal:6 [2022-10-25 23:22:08.359494] --log-level=lib.cryptodev:5 [2022-10-25 23:22:08.359501] --log-level=user1:6 [2022-10-25 23:22:08.359511] --iova-mode=pa [2022-10-25 23:22:08.359519] --base-virtaddr=0x200000000000 [2022-10-25 23:22:08.359527] --match-allocations [2022-10-25 23:22:08.359535] --file-prefix=spdk_pid84265 [2022-10-25 23:22:08.359542] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:22:08.497812] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:22:08.545510] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:22:08.546171] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 223763.55 IOPS 874.08 MiB/s
=============================================================
Total : 223763.55 IOPS 874.08 MiB/s
[2022-10-25 23:22:21.359973] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:22:21.360044] [ DPDK EAL parameters: [2022-10-25 23:22:21.360054] bdevperf [2022-10-25 23:22:21.360063] --no-shconf [2022-10-25 23:22:21.360071] -l 0 [2022-10-25 23:22:21.360079] --huge-unlink [2022-10-25 23:22:21.360087] --log-level=lib.eal:6 [2022-10-25 23:22:21.360095] --log-level=lib.cryptodev:5 [2022-10-25 23:22:21.360103] --log-level=user1:6 [2022-10-25 23:22:21.360110] --iova-mode=pa [2022-10-25 23:22:21.360118] --base-virtaddr=0x200000000000 [2022-10-25 23:22:21.360126] --match-allocations [2022-10-25 23:22:21.360134] --file-prefix=spdk_pid84290 [2022-10-25 23:22:21.360141] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:22:21.498321] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:22:21.545611] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:22:21.546279] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 223690.52 IOPS 873.79 MiB/s
=============================================================
Total : 223690.52 IOPS 873.79 MiB/s
[2022-10-25 23:22:34.349231] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:22:34.349305] [ DPDK EAL parameters: [2022-10-25 23:22:34.349316] bdevperf [2022-10-25 23:22:34.349325] --no-shconf [2022-10-25 23:22:34.349333] -l 0 [2022-10-25 23:22:34.349342] --huge-unlink [2022-10-25 23:22:34.349350] --log-level=lib.eal:6 [2022-10-25 23:22:34.349358] --log-level=lib.cryptodev:5 [2022-10-25 23:22:34.349365] --log-level=user1:6 [2022-10-25 23:22:34.349374] --iova-mode=pa [2022-10-25 23:22:34.349383] --base-virtaddr=0x200000000000 [2022-10-25 23:22:34.349391] --match-allocations [2022-10-25 23:22:34.349399] --file-prefix=spdk_pid84315 [2022-10-25 23:22:34.349407] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:22:34.487798] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:22:34.529488] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:22:34.530150] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 223783.71 IOPS 874.16 MiB/s
=============================================================
Total : 223783.71 IOPS 874.16 MiB/s
[2022-10-25 23:22:47.374691] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:22:47.374761] [ DPDK EAL parameters: [2022-10-25 23:22:47.374772] bdevperf [2022-10-25 23:22:47.374781] --no-shconf [2022-10-25 23:22:47.374789] -l 0 [2022-10-25 23:22:47.374797] --huge-unlink [2022-10-25 23:22:47.374806] --log-level=lib.eal:6 [2022-10-25 23:22:47.374814] --log-level=lib.cryptodev:5 [2022-10-25 23:22:47.374821] --log-level=user1:6 [2022-10-25 23:22:47.374847] --iova-mode=pa [2022-10-25 23:22:47.374857] --base-virtaddr=0x200000000000 [2022-10-25 23:22:47.374866] --match-allocations [2022-10-25 23:22:47.374873] --file-prefix=spdk_pid84340 [2022-10-25 23:22:47.374887] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:22:47.513230] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:22:47.553617] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:22:47.554293] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 87084.15 IOPS 340.17 MiB/s
=============================================================
Total : 87084.15 IOPS 340.17 MiB/s
[2022-10-25 23:04:52.128230] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:04:52.128295] [ DPDK EAL parameters: [2022-10-25 23:04:52.128306] bdevperf [2022-10-25 23:04:52.128315] --no-shconf [2022-10-25 23:04:52.128323] -l 0 [2022-10-25 23:04:52.128331] --huge-unlink [2022-10-25 23:04:52.128339] --log-level=lib.eal:6 [2022-10-25 23:04:52.128347] --log-level=lib.cryptodev:5 [2022-10-25 23:04:52.128355] --log-level=user1:6 [2022-10-25 23:04:52.128362] --iova-mode=pa [2022-10-25 23:04:52.128371] --base-virtaddr=0x200000000000 [2022-10-25 23:04:52.128379] --match-allocations [2022-10-25 23:04:52.128386] --file-prefix=spdk_pid82990 [2022-10-25 23:04:52.128394] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:04:52.266924] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:04:52.325526] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:04:52.326204] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 223745.85 IOPS 874.01 MiB/s
=============================================================
Total : 223745.85 IOPS 874.01 MiB/s
[2022-10-25 23:23:00.358689] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:23:00.358758] [ DPDK EAL parameters: [2022-10-25 23:23:00.358768] bdevperf [2022-10-25 23:23:00.358777] --no-shconf [2022-10-25 23:23:00.358786] -l 0 [2022-10-25 23:23:00.358795] --huge-unlink [2022-10-25 23:23:00.358803] --log-level=lib.eal:6 [2022-10-25 23:23:00.358811] --log-level=lib.cryptodev:5 [2022-10-25 23:23:00.358819] --log-level=user1:6 [2022-10-25 23:23:00.358826] --iova-mode=pa [2022-10-25 23:23:00.358834] --base-virtaddr=0x200000000000 [2022-10-25 23:23:00.358842] --match-allocations [2022-10-25 23:23:00.358850] --file-prefix=spdk_pid84365 [2022-10-25 23:23:00.358858] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:23:00.497334] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:23:00.545520] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:23:00.546189] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
rm: cannot remove '/tmp/spdk_bdev_xnvme_io_uring.conf': No such file or directory
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 95399.03 IOPS 372.65 MiB/s
=============================================================
Total : 95399.03 IOPS 372.65 MiB/s
[2022-10-25 23:05:05.060149] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:05:05.060219] [ DPDK EAL parameters: [2022-10-25 23:05:05.060231] bdevperf [2022-10-25 23:05:05.060240] --no-shconf [2022-10-25 23:05:05.060249] -l 0 [2022-10-25 23:05:05.060257] --huge-unlink [2022-10-25 23:05:05.060265] --log-level=lib.eal:6 [2022-10-25 23:05:05.060273] --log-level=lib.cryptodev:5 [2022-10-25 23:05:05.060280] --log-level=user1:6 [2022-10-25 23:05:05.060288] --iova-mode=pa [2022-10-25 23:05:05.060296] --base-virtaddr=0x200000000000 [2022-10-25 23:05:05.060304] --match-allocations [2022-10-25 23:05:05.060312] --file-prefix=spdk_pid83016 [2022-10-25 23:05:05.060321] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:05:05.198531] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:05:05.245527] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:05:05.246196] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 95115.58 IOPS 371.55 MiB/s
=============================================================
Total : 95115.58 IOPS 371.55 MiB/s
[2022-10-25 23:05:17.970774] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:05:17.970842] [ DPDK EAL parameters: [2022-10-25 23:05:17.970852] bdevperf [2022-10-25 23:05:17.970861] --no-shconf [2022-10-25 23:05:17.970869] -l 0 [2022-10-25 23:05:17.970877] --huge-unlink [2022-10-25 23:05:17.970886] --log-level=lib.eal:6 [2022-10-25 23:05:17.970894] --log-level=lib.cryptodev:5 [2022-10-25 23:05:17.970902] --log-level=user1:6 [2022-10-25 23:05:17.970910] --iova-mode=pa [2022-10-25 23:05:17.970918] --base-virtaddr=0x200000000000 [2022-10-25 23:05:17.970926] --match-allocations [2022-10-25 23:05:17.970934] --file-prefix=spdk_pid83041 [2022-10-25 23:05:17.970943] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:05:18.109398] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:05:18.153610] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:05:18.154319] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 95206.38 IOPS 371.90 MiB/s
=============================================================
Total : 95206.38 IOPS 371.90 MiB/s
[2022-10-25 23:05:30.898804] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:05:30.898875] [ DPDK EAL parameters: [2022-10-25 23:05:30.898887] bdevperf [2022-10-25 23:05:30.898896] --no-shconf [2022-10-25 23:05:30.898905] -l 0 [2022-10-25 23:05:30.898912] --huge-unlink [2022-10-25 23:05:30.898921] --log-level=lib.eal:6 [2022-10-25 23:05:30.898929] --log-level=lib.cryptodev:5 [2022-10-25 23:05:30.898937] --log-level=user1:6 [2022-10-25 23:05:30.898945] --iova-mode=pa [2022-10-25 23:05:30.898953] --base-virtaddr=0x200000000000 [2022-10-25 23:05:30.898960] --match-allocations [2022-10-25 23:05:30.898968] --file-prefix=spdk_pid83066 [2022-10-25 23:05:30.898976] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:05:31.037583] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:05:31.081613] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:05:31.082310] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
rm: cannot remove '/tmp/spdk_bdev_aio_foo.conf': No such file or directory
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 94374.27 IOPS 368.65 MiB/s
=============================================================
Total : 94374.27 IOPS 368.65 MiB/s
[2022-10-25 23:05:43.831772] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:05:43.831843] [ DPDK EAL parameters: [2022-10-25 23:05:43.831854] bdevperf [2022-10-25 23:05:43.831863] --no-shconf [2022-10-25 23:05:43.831872] -l 0 [2022-10-25 23:05:43.831880] --huge-unlink [2022-10-25 23:05:43.831888] --log-level=lib.eal:6 [2022-10-25 23:05:43.831897] --log-level=lib.cryptodev:5 [2022-10-25 23:05:43.831905] --log-level=user1:6 [2022-10-25 23:05:43.831913] --iova-mode=pa [2022-10-25 23:05:43.831921] --base-virtaddr=0x200000000000 [2022-10-25 23:05:43.831929] --match-allocations [2022-10-25 23:05:43.831937] --file-prefix=spdk_pid83091 [2022-10-25 23:05:43.831946] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:05:43.970469] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:05:44.009497] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:05:44.010185] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 94300.13 IOPS 368.36 MiB/s
=============================================================
Total : 94300.13 IOPS 368.36 MiB/s
[2022-10-25 23:05:56.804496] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:05:56.804569] [ DPDK EAL parameters: [2022-10-25 23:05:56.804581] bdevperf [2022-10-25 23:05:56.804590] --no-shconf [2022-10-25 23:05:56.804599] -l 0 [2022-10-25 23:05:56.804608] --huge-unlink [2022-10-25 23:05:56.804617] --log-level=lib.eal:6 [2022-10-25 23:05:56.804625] --log-level=lib.cryptodev:5 [2022-10-25 23:05:56.804634] --log-level=user1:6 [2022-10-25 23:05:56.804643] --iova-mode=pa [2022-10-25 23:05:56.804652] --base-virtaddr=0x200000000000 [2022-10-25 23:05:56.804660] --match-allocations [2022-10-25 23:05:56.804670] --file-prefix=spdk_pid83116 [2022-10-25 23:05:56.804679] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:05:56.943818] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:05:56.989578] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:05:56.990266] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 94145.55 IOPS 367.76 MiB/s
=============================================================
Total : 94145.55 IOPS 367.76 MiB/s
[2022-10-25 23:06:09.798113] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:06:09.798184] [ DPDK EAL parameters: [2022-10-25 23:06:09.798194] bdevperf [2022-10-25 23:06:09.798203] --no-shconf [2022-10-25 23:06:09.798211] -l 0 [2022-10-25 23:06:09.798219] --huge-unlink [2022-10-25 23:06:09.798228] --log-level=lib.eal:6 [2022-10-25 23:06:09.798235] --log-level=lib.cryptodev:5 [2022-10-25 23:06:09.798243] --log-level=user1:6 [2022-10-25 23:06:09.798251] --iova-mode=pa [2022-10-25 23:06:09.798259] --base-virtaddr=0x200000000000 [2022-10-25 23:06:09.798267] --match-allocations [2022-10-25 23:06:09.798274] --file-prefix=spdk_pid83141 [2022-10-25 23:06:09.798282] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:06:09.937122] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:06:09.989458] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:06:09.990130] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
rm: cannot remove '/tmp/spdk_bdev_xnvme_libaio.conf': No such file or directory
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 92010.24 IOPS 359.42 MiB/s
=============================================================
Total : 92010.24 IOPS 359.42 MiB/s
[2022-10-25 23:06:22.801816] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:06:22.801884] [ DPDK EAL parameters: [2022-10-25 23:06:22.801895] bdevperf [2022-10-25 23:06:22.801904] --no-shconf [2022-10-25 23:06:22.801913] -l 0 [2022-10-25 23:06:22.801920] --huge-unlink [2022-10-25 23:06:22.801928] --log-level=lib.eal:6 [2022-10-25 23:06:22.801936] --log-level=lib.cryptodev:5 [2022-10-25 23:06:22.801943] --log-level=user1:6 [2022-10-25 23:06:22.801968] --iova-mode=pa [2022-10-25 23:06:22.801976] --base-virtaddr=0x200000000000 [2022-10-25 23:06:22.801983] --match-allocations [2022-10-25 23:06:22.801990] --file-prefix=spdk_pid83166 [2022-10-25 23:06:22.802006] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:06:22.940465] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:06:22.981534] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:06:22.982224] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 92095.63 IOPS 359.75 MiB/s
=============================================================
Total : 92095.63 IOPS 359.75 MiB/s
[2022-10-25 23:06:35.798932] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:06:35.799005] [ DPDK EAL parameters: [2022-10-25 23:06:35.799016] bdevperf [2022-10-25 23:06:35.799025] --no-shconf [2022-10-25 23:06:35.799033] -l 0 [2022-10-25 23:06:35.799041] --huge-unlink [2022-10-25 23:06:35.799050] --log-level=lib.eal:6 [2022-10-25 23:06:35.799058] --log-level=lib.cryptodev:5 [2022-10-25 23:06:35.799065] --log-level=user1:6 [2022-10-25 23:06:35.799075] --iova-mode=pa [2022-10-25 23:06:35.799082] --base-virtaddr=0x200000000000 [2022-10-25 23:06:35.799091] --match-allocations [2022-10-25 23:06:35.799099] --file-prefix=spdk_pid83193 [2022-10-25 23:06:35.799107] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:06:35.937685] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:06:35.993482] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:06:35.994154] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 91701.04 IOPS 358.21 MiB/s
=============================================================
Total : 91701.04 IOPS 358.21 MiB/s
[2022-10-25 23:06:48.806875] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:06:48.806943] [ DPDK EAL parameters: [2022-10-25 23:06:48.806953] bdevperf [2022-10-25 23:06:48.806962] --no-shconf [2022-10-25 23:06:48.806970] -l 0 [2022-10-25 23:06:48.806977] --huge-unlink [2022-10-25 23:06:48.806985] --log-level=lib.eal:6 [2022-10-25 23:06:48.806993] --log-level=lib.cryptodev:5 [2022-10-25 23:06:48.807000] --log-level=user1:6 [2022-10-25 23:06:48.807025] --iova-mode=pa [2022-10-25 23:06:48.807033] --base-virtaddr=0x200000000000 [2022-10-25 23:06:48.807041] --match-allocations [2022-10-25 23:06:48.807049] --file-prefix=spdk_pid83218 [2022-10-25 23:06:48.807064] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:06:48.945250] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:06:48.985598] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:06:48.986291] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 137677.93 IOPS 537.80 MiB/s
=============================================================
Total : 137677.93 IOPS 537.80 MiB/s
[2022-10-25 23:07:01.817511] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:07:01.817582] [ DPDK EAL parameters: [2022-10-25 23:07:01.817593] bdevperf [2022-10-25 23:07:01.817602] --no-shconf [2022-10-25 23:07:01.817611] -l 0 [2022-10-25 23:07:01.817618] --huge-unlink [2022-10-25 23:07:01.817628] --log-level=lib.eal:6 [2022-10-25 23:07:01.817635] --log-level=lib.cryptodev:5 [2022-10-25 23:07:01.817643] --log-level=user1:6 [2022-10-25 23:07:01.817651] --iova-mode=pa [2022-10-25 23:07:01.817659] --base-virtaddr=0x200000000000 [2022-10-25 23:07:01.817667] --match-allocations [2022-10-25 23:07:01.817675] --file-prefix=spdk_pid83243 [2022-10-25 23:07:01.817684] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:07:01.956216] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:07:02.001505] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:07:02.002177] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 137638.53 IOPS 537.65 MiB/s
=============================================================
Total : 137638.53 IOPS 537.65 MiB/s
[2022-10-25 23:07:14.742705] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:07:14.742773] [ DPDK EAL parameters: [2022-10-25 23:07:14.742783] bdevperf [2022-10-25 23:07:14.742792] --no-shconf [2022-10-25 23:07:14.742800] -l 0 [2022-10-25 23:07:14.742808] --huge-unlink [2022-10-25 23:07:14.742815] --log-level=lib.eal:6 [2022-10-25 23:07:14.742822] --log-level=lib.cryptodev:5 [2022-10-25 23:07:14.742829] --log-level=user1:6 [2022-10-25 23:07:14.742837] --iova-mode=pa [2022-10-25 23:07:14.742844] --base-virtaddr=0x200000000000 [2022-10-25 23:07:14.742851] --match-allocations [2022-10-25 23:07:14.742858] --file-prefix=spdk_pid83268 [2022-10-25 23:07:14.742866] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:07:14.881095] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:07:14.929550] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:07:14.930229] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 122234.74 IOPS 477.48 MiB/s
=============================================================
Total : 122234.74 IOPS 477.48 MiB/s
[2022-10-25 23:07:27.731543] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:07:27.731614] [ DPDK EAL parameters: [2022-10-25 23:07:27.731625] bdevperf [2022-10-25 23:07:27.731633] --no-shconf [2022-10-25 23:07:27.731642] -l 0 [2022-10-25 23:07:27.731650] --huge-unlink [2022-10-25 23:07:27.731659] --log-level=lib.eal:6 [2022-10-25 23:07:27.731666] --log-level=lib.cryptodev:5 [2022-10-25 23:07:27.731673] --log-level=user1:6 [2022-10-25 23:07:27.731681] --iova-mode=pa [2022-10-25 23:07:27.731689] --base-virtaddr=0x200000000000 [2022-10-25 23:07:27.731697] --match-allocations [2022-10-25 23:07:27.731705] --file-prefix=spdk_pid83293 [2022-10-25 23:07:27.731713] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:07:27.870333] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:07:27.917497] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:07:27.918172] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 123370.19 IOPS 481.91 MiB/s
=============================================================
Total : 123370.19 IOPS 481.91 MiB/s
[2022-10-25 23:07:40.684960] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:07:40.685028] [ DPDK EAL parameters: [2022-10-25 23:07:40.685039] bdevperf [2022-10-25 23:07:40.685048] --no-shconf [2022-10-25 23:07:40.685057] -l 0 [2022-10-25 23:07:40.685065] --huge-unlink [2022-10-25 23:07:40.685072] --log-level=lib.eal:6 [2022-10-25 23:07:40.685081] --log-level=lib.cryptodev:5 [2022-10-25 23:07:40.685088] --log-level=user1:6 [2022-10-25 23:07:40.685096] --iova-mode=pa [2022-10-25 23:07:40.685103] --base-virtaddr=0x200000000000 [2022-10-25 23:07:40.685112] --match-allocations [2022-10-25 23:07:40.685120] --file-prefix=spdk_pid83318 [2022-10-25 23:07:40.685128] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:07:40.823519] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:07:40.865686] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:07:40.866354] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 123229.34 IOPS 481.36 MiB/s
=============================================================
Total : 123229.34 IOPS 481.36 MiB/s
[2022-10-25 23:07:53.604306] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:07:53.604381] [ DPDK EAL parameters: [2022-10-25 23:07:53.604392] bdevperf [2022-10-25 23:07:53.604401] --no-shconf [2022-10-25 23:07:53.604410] -l 0 [2022-10-25 23:07:53.604418] --huge-unlink [2022-10-25 23:07:53.604427] --log-level=lib.eal:6 [2022-10-25 23:07:53.604434] --log-level=lib.cryptodev:5 [2022-10-25 23:07:53.604442] --log-level=user1:6 [2022-10-25 23:07:53.604450] --iova-mode=pa [2022-10-25 23:07:53.604458] --base-virtaddr=0x200000000000 [2022-10-25 23:07:53.604466] --match-allocations [2022-10-25 23:07:53.604474] --file-prefix=spdk_pid83343 [2022-10-25 23:07:53.604482] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:07:53.742580] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:07:53.813506] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:07:53.814176] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 123447.21 IOPS 482.22 MiB/s
=============================================================
Total : 123447.21 IOPS 482.22 MiB/s
[2022-10-25 23:08:06.563163] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:08:06.563231] [ DPDK EAL parameters: [2022-10-25 23:08:06.563241] bdevperf [2022-10-25 23:08:06.563250] --no-shconf [2022-10-25 23:08:06.563258] -l 0 [2022-10-25 23:08:06.563267] --huge-unlink [2022-10-25 23:08:06.563276] --log-level=lib.eal:6 [2022-10-25 23:08:06.563283] --log-level=lib.cryptodev:5 [2022-10-25 23:08:06.563291] --log-level=user1:6 [2022-10-25 23:08:06.563298] --iova-mode=pa [2022-10-25 23:08:06.563307] --base-virtaddr=0x200000000000 [2022-10-25 23:08:06.563315] --match-allocations [2022-10-25 23:08:06.563323] --file-prefix=spdk_pid83368 [2022-10-25 23:08:06.563331] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:08:06.701480] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:08:06.761538] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:08:06.762219] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 132739.00 IOPS 518.51 MiB/s
=============================================================
Total : 132739.00 IOPS 518.51 MiB/s
[2022-10-25 23:08:19.508671] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:08:19.508741] [ DPDK EAL parameters: [2022-10-25 23:08:19.508752] bdevperf [2022-10-25 23:08:19.508761] --no-shconf [2022-10-25 23:08:19.508770] -l 0 [2022-10-25 23:08:19.508778] --huge-unlink [2022-10-25 23:08:19.508786] --log-level=lib.eal:6 [2022-10-25 23:08:19.508793] --log-level=lib.cryptodev:5 [2022-10-25 23:08:19.508801] --log-level=user1:6 [2022-10-25 23:08:19.508808] --iova-mode=pa [2022-10-25 23:08:19.508816] --base-virtaddr=0x200000000000 [2022-10-25 23:08:19.508824] --match-allocations [2022-10-25 23:08:19.508832] --file-prefix=spdk_pid83393 [2022-10-25 23:08:19.508840] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:08:19.647435] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:08:19.689565] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:08:19.690233] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 132646.77 IOPS 518.15 MiB/s
=============================================================
Total : 132646.77 IOPS 518.15 MiB/s
[2022-10-25 23:08:32.474289] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:08:32.474360] [ DPDK EAL parameters: [2022-10-25 23:08:32.474370] bdevperf [2022-10-25 23:08:32.474379] --no-shconf [2022-10-25 23:08:32.474389] -l 0 [2022-10-25 23:08:32.474397] --huge-unlink [2022-10-25 23:08:32.474404] --log-level=lib.eal:6 [2022-10-25 23:08:32.474412] --log-level=lib.cryptodev:5 [2022-10-25 23:08:32.474419] --log-level=user1:6 [2022-10-25 23:08:32.474426] --iova-mode=pa [2022-10-25 23:08:32.474434] --base-virtaddr=0x200000000000 [2022-10-25 23:08:32.474442] --match-allocations [2022-10-25 23:08:32.474451] --file-prefix=spdk_pid83419 [2022-10-25 23:08:32.474458] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:08:32.612935] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:08:32.661606] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:08:32.662286] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 132765.93 IOPS 518.62 MiB/s
=============================================================
Total : 132765.93 IOPS 518.62 MiB/s
[2022-10-25 23:08:45.526319] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:08:45.526390] [ DPDK EAL parameters: [2022-10-25 23:08:45.526400] bdevperf [2022-10-25 23:08:45.526409] --no-shconf [2022-10-25 23:08:45.526418] -l 0 [2022-10-25 23:08:45.526426] --huge-unlink [2022-10-25 23:08:45.526433] --log-level=lib.eal:6 [2022-10-25 23:08:45.526441] --log-level=lib.cryptodev:5 [2022-10-25 23:08:45.526448] --log-level=user1:6 [2022-10-25 23:08:45.526456] --iova-mode=pa [2022-10-25 23:08:45.526464] --base-virtaddr=0x200000000000 [2022-10-25 23:08:45.526472] --match-allocations [2022-10-25 23:08:45.526479] --file-prefix=spdk_pid83445 [2022-10-25 23:08:45.526487] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:08:45.665426] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:08:45.713532] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:08:45.714200] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 131267.29 IOPS 512.76 MiB/s
=============================================================
Total : 131267.29 IOPS 512.76 MiB/s
[2022-10-25 23:08:58.466029] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:08:58.466097] [ DPDK EAL parameters: [2022-10-25 23:08:58.466108] bdevperf [2022-10-25 23:08:58.466116] --no-shconf [2022-10-25 23:08:58.466125] -l 0 [2022-10-25 23:08:58.466133] --huge-unlink [2022-10-25 23:08:58.466142] --log-level=lib.eal:6 [2022-10-25 23:08:58.466149] --log-level=lib.cryptodev:5 [2022-10-25 23:08:58.466157] --log-level=user1:6 [2022-10-25 23:08:58.466172] --iova-mode=pa [2022-10-25 23:08:58.466180] --base-virtaddr=0x200000000000 [2022-10-25 23:08:58.466189] --match-allocations [2022-10-25 23:08:58.466197] --file-prefix=spdk_pid83471 [2022-10-25 23:08:58.466205] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:08:58.604528] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:08:58.657517] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:08:58.658187] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 131511.69 IOPS 513.72 MiB/s
=============================================================
Total : 131511.69 IOPS 513.72 MiB/s
[2022-10-25 23:09:11.464489] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:09:11.464557] [ DPDK EAL parameters: [2022-10-25 23:09:11.464568] bdevperf [2022-10-25 23:09:11.464577] --no-shconf [2022-10-25 23:09:11.464586] -l 0 [2022-10-25 23:09:11.464594] --huge-unlink [2022-10-25 23:09:11.464602] --log-level=lib.eal:6 [2022-10-25 23:09:11.464611] --log-level=lib.cryptodev:5 [2022-10-25 23:09:11.464618] --log-level=user1:6 [2022-10-25 23:09:11.464626] --iova-mode=pa [2022-10-25 23:09:11.464634] --base-virtaddr=0x200000000000 [2022-10-25 23:09:11.464642] --match-allocations [2022-10-25 23:09:11.464650] --file-prefix=spdk_pid83497 [2022-10-25 23:09:11.464658] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:09:11.603992] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:09:11.645517] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:09:11.646198] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 131606.51 IOPS 514.09 MiB/s
=============================================================
Total : 131606.51 IOPS 514.09 MiB/s
[2022-10-25 23:09:24.448368] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:09:24.448436] [ DPDK EAL parameters: [2022-10-25 23:09:24.448447] bdevperf [2022-10-25 23:09:24.448456] --no-shconf [2022-10-25 23:09:24.448465] -l 0 [2022-10-25 23:09:24.448473] --huge-unlink [2022-10-25 23:09:24.448481] --log-level=lib.eal:6 [2022-10-25 23:09:24.448489] --log-level=lib.cryptodev:5 [2022-10-25 23:09:24.448496] --log-level=user1:6 [2022-10-25 23:09:24.448503] --iova-mode=pa [2022-10-25 23:09:24.448511] --base-virtaddr=0x200000000000 [2022-10-25 23:09:24.448520] --match-allocations [2022-10-25 23:09:24.448527] --file-prefix=spdk_pid83522 [2022-10-25 23:09:24.448535] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:09:24.587595] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:09:24.633513] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:09:24.634174] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 128907.87 IOPS 503.55 MiB/s
=============================================================
Total : 128907.87 IOPS 503.55 MiB/s
[2022-10-25 23:09:37.447829] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:09:37.447899] [ DPDK EAL parameters: [2022-10-25 23:09:37.447909] bdevperf [2022-10-25 23:09:37.447918] --no-shconf [2022-10-25 23:09:37.447926] -l 0 [2022-10-25 23:09:37.447934] --huge-unlink [2022-10-25 23:09:37.447943] --log-level=lib.eal:6 [2022-10-25 23:09:37.447951] --log-level=lib.cryptodev:5 [2022-10-25 23:09:37.447958] --log-level=user1:6 [2022-10-25 23:09:37.447965] --iova-mode=pa [2022-10-25 23:09:37.447973] --base-virtaddr=0x200000000000 [2022-10-25 23:09:37.447981] --match-allocations [2022-10-25 23:09:37.447988] --file-prefix=spdk_pid83547 [2022-10-25 23:09:37.447996] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:09:37.586338] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:09:37.633485] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:09:37.634187] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 129471.51 IOPS 505.75 MiB/s
=============================================================
Total : 129471.51 IOPS 505.75 MiB/s
[2022-10-25 23:09:50.417714] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:09:50.417786] [ DPDK EAL parameters: [2022-10-25 23:09:50.417797] bdevperf [2022-10-25 23:09:50.417806] --no-shconf [2022-10-25 23:09:50.417815] -l 0 [2022-10-25 23:09:50.417823] --huge-unlink [2022-10-25 23:09:50.417830] --log-level=lib.eal:6 [2022-10-25 23:09:50.417838] --log-level=lib.cryptodev:5 [2022-10-25 23:09:50.417845] --log-level=user1:6 [2022-10-25 23:09:50.417855] --iova-mode=pa [2022-10-25 23:09:50.417864] --base-virtaddr=0x200000000000 [2022-10-25 23:09:50.417872] --match-allocations [2022-10-25 23:09:50.417880] --file-prefix=spdk_pid83574 [2022-10-25 23:09:50.417889] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:09:50.556324] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:09:50.597526] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:09:50.598215] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 134840.52 IOPS 526.72 MiB/s
=============================================================
Total : 134840.52 IOPS 526.72 MiB/s
[2022-10-25 23:10:03.412061] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:10:03.412135] [ DPDK EAL parameters: [2022-10-25 23:10:03.412146] bdevperf [2022-10-25 23:10:03.412155] --no-shconf [2022-10-25 23:10:03.412164] -l 0 [2022-10-25 23:10:03.412171] --huge-unlink [2022-10-25 23:10:03.412179] --log-level=lib.eal:6 [2022-10-25 23:10:03.412187] --log-level=lib.cryptodev:5 [2022-10-25 23:10:03.412195] --log-level=user1:6 [2022-10-25 23:10:03.412204] --iova-mode=pa [2022-10-25 23:10:03.412212] --base-virtaddr=0x200000000000 [2022-10-25 23:10:03.412220] --match-allocations [2022-10-25 23:10:03.412227] --file-prefix=spdk_pid83599 [2022-10-25 23:10:03.412236] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:10:03.550916] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:10:03.597661] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:10:03.598365] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 220666.80 IOPS 861.98 MiB/s
=============================================================
Total : 220666.80 IOPS 861.98 MiB/s
[2022-10-25 23:10:16.626846] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:10:16.626916] [ DPDK EAL parameters: [2022-10-25 23:10:16.626927] bdevperf [2022-10-25 23:10:16.626935] --no-shconf [2022-10-25 23:10:16.626943] -l 0 [2022-10-25 23:10:16.626951] --huge-unlink [2022-10-25 23:10:16.626958] --log-level=lib.eal:6 [2022-10-25 23:10:16.626966] --log-level=lib.cryptodev:5 [2022-10-25 23:10:16.626973] --log-level=user1:6 [2022-10-25 23:10:16.626980] --iova-mode=pa [2022-10-25 23:10:16.626987] --base-virtaddr=0x200000000000 [2022-10-25 23:10:16.626995] --match-allocations [2022-10-25 23:10:16.627002] --file-prefix=spdk_pid83624 [2022-10-25 23:10:16.627009] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:10:16.765360] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:10:16.817512] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:10:16.818222] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 216380.02 IOPS 845.23 MiB/s
=============================================================
Total : 216380.02 IOPS 845.23 MiB/s
[2022-10-25 23:10:29.570713] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:10:29.570782] [ DPDK EAL parameters: [2022-10-25 23:10:29.570792] bdevperf [2022-10-25 23:10:29.570801] --no-shconf [2022-10-25 23:10:29.570810] -l 0 [2022-10-25 23:10:29.570818] --huge-unlink [2022-10-25 23:10:29.570826] --log-level=lib.eal:6 [2022-10-25 23:10:29.570834] --log-level=lib.cryptodev:5 [2022-10-25 23:10:29.570842] --log-level=user1:6 [2022-10-25 23:10:29.570850] --iova-mode=pa [2022-10-25 23:10:29.570859] --base-virtaddr=0x200000000000 [2022-10-25 23:10:29.570867] --match-allocations [2022-10-25 23:10:29.570876] --file-prefix=spdk_pid83649 [2022-10-25 23:10:29.570884] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:10:29.709427] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:10:29.753600] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:10:29.754303] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 216607.90 IOPS 846.12 MiB/s
=============================================================
Total : 216607.90 IOPS 846.12 MiB/s
[2022-10-25 23:10:42.527926] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:10:42.527995] [ DPDK EAL parameters: [2022-10-25 23:10:42.528006] bdevperf [2022-10-25 23:10:42.528015] --no-shconf [2022-10-25 23:10:42.528023] -l 0 [2022-10-25 23:10:42.528030] --huge-unlink [2022-10-25 23:10:42.528039] --log-level=lib.eal:6 [2022-10-25 23:10:42.528047] --log-level=lib.cryptodev:5 [2022-10-25 23:10:42.528054] --log-level=user1:6 [2022-10-25 23:10:42.528080] --iova-mode=pa [2022-10-25 23:10:42.528088] --base-virtaddr=0x200000000000 [2022-10-25 23:10:42.528096] --match-allocations [2022-10-25 23:10:42.528104] --file-prefix=spdk_pid83674 [2022-10-25 23:10:42.528119] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:10:42.666440] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:10:42.709521] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:10:42.710222] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 209760.64 IOPS 819.38 MiB/s
=============================================================
Total : 209760.64 IOPS 819.38 MiB/s
[2022-10-25 23:10:55.470262] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:10:55.470332] [ DPDK EAL parameters: [2022-10-25 23:10:55.470343] bdevperf [2022-10-25 23:10:55.470352] --no-shconf [2022-10-25 23:10:55.470361] -l 0 [2022-10-25 23:10:55.470369] --huge-unlink [2022-10-25 23:10:55.470378] --log-level=lib.eal:6 [2022-10-25 23:10:55.470386] --log-level=lib.cryptodev:5 [2022-10-25 23:10:55.470393] --log-level=user1:6 [2022-10-25 23:10:55.470402] --iova-mode=pa [2022-10-25 23:10:55.470409] --base-virtaddr=0x200000000000 [2022-10-25 23:10:55.470417] --match-allocations [2022-10-25 23:10:55.470425] --file-prefix=spdk_pid83699 [2022-10-25 23:10:55.470433] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:10:55.609173] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:10:55.669534] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:10:55.670217] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 209376.05 IOPS 817.88 MiB/s
=============================================================
Total : 209376.05 IOPS 817.88 MiB/s
[2022-10-25 23:11:08.404264] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:11:08.404334] [ DPDK EAL parameters: [2022-10-25 23:11:08.404345] bdevperf [2022-10-25 23:11:08.404354] --no-shconf [2022-10-25 23:11:08.404362] -l 0 [2022-10-25 23:11:08.404370] --huge-unlink [2022-10-25 23:11:08.404379] --log-level=lib.eal:6 [2022-10-25 23:11:08.404386] --log-level=lib.cryptodev:5 [2022-10-25 23:11:08.404394] --log-level=user1:6 [2022-10-25 23:11:08.404402] --iova-mode=pa [2022-10-25 23:11:08.404410] --base-virtaddr=0x200000000000 [2022-10-25 23:11:08.404418] --match-allocations [2022-10-25 23:11:08.404426] --file-prefix=spdk_pid83724 [2022-10-25 23:11:08.404434] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:11:08.542984] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:11:08.589571] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:11:08.590268] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 209107.46 IOPS 816.83 MiB/s
=============================================================
Total : 209107.46 IOPS 816.83 MiB/s
[2022-10-25 23:11:21.347093] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:11:21.347171] [ DPDK EAL parameters: [2022-10-25 23:11:21.347182] bdevperf [2022-10-25 23:11:21.347191] --no-shconf [2022-10-25 23:11:21.347200] -l 0 [2022-10-25 23:11:21.347208] --huge-unlink [2022-10-25 23:11:21.347217] --log-level=lib.eal:6 [2022-10-25 23:11:21.347225] --log-level=lib.cryptodev:5 [2022-10-25 23:11:21.347234] --log-level=user1:6 [2022-10-25 23:11:21.347242] --iova-mode=pa [2022-10-25 23:11:21.347250] --base-virtaddr=0x200000000000 [2022-10-25 23:11:21.347258] --match-allocations [2022-10-25 23:11:21.347266] --file-prefix=spdk_pid83749 [2022-10-25 23:11:21.347274] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:11:21.485620] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:11:21.541528] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:11:21.542206] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 221094.82 IOPS 863.65 MiB/s
=============================================================
Total : 221094.82 IOPS 863.65 MiB/s
[2022-10-25 23:11:34.290214] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:11:34.290284] [ DPDK EAL parameters: [2022-10-25 23:11:34.290294] bdevperf [2022-10-25 23:11:34.290303] --no-shconf [2022-10-25 23:11:34.290311] -l 0 [2022-10-25 23:11:34.290319] --huge-unlink [2022-10-25 23:11:34.290326] --log-level=lib.eal:6 [2022-10-25 23:11:34.290335] --log-level=lib.cryptodev:5 [2022-10-25 23:11:34.290342] --log-level=user1:6 [2022-10-25 23:11:34.290350] --iova-mode=pa [2022-10-25 23:11:34.290376] --base-virtaddr=0x200000000000 [2022-10-25 23:11:34.290385] --match-allocations [2022-10-25 23:11:34.290393] --file-prefix=spdk_pid83774 [2022-10-25 23:11:34.290409] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:11:34.428902] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:11:34.469602] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:11:34.470285] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 221310.00 IOPS 864.49 MiB/s
=============================================================
Total : 221310.00 IOPS 864.49 MiB/s
[2022-10-25 23:11:47.217369] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:11:47.217440] [ DPDK EAL parameters: [2022-10-25 23:11:47.217451] bdevperf [2022-10-25 23:11:47.217461] --no-shconf [2022-10-25 23:11:47.217468] -l 0 [2022-10-25 23:11:47.217476] --huge-unlink [2022-10-25 23:11:47.217484] --log-level=lib.eal:6 [2022-10-25 23:11:47.217491] --log-level=lib.cryptodev:5 [2022-10-25 23:11:47.217498] --log-level=user1:6 [2022-10-25 23:11:47.217505] --iova-mode=pa [2022-10-25 23:11:47.217513] --base-virtaddr=0x200000000000 [2022-10-25 23:11:47.217520] --match-allocations [2022-10-25 23:11:47.217527] --file-prefix=spdk_pid83799 [2022-10-25 23:11:47.217534] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:11:47.355898] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:11:47.397531] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:11:47.398212] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 221270.62 IOPS 864.34 MiB/s
=============================================================
Total : 221270.62 IOPS 864.34 MiB/s
[2022-10-25 23:12:00.149272] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:12:00.149341] [ DPDK EAL parameters: [2022-10-25 23:12:00.149351] bdevperf [2022-10-25 23:12:00.149360] --no-shconf [2022-10-25 23:12:00.149369] -l 0 [2022-10-25 23:12:00.149377] --huge-unlink [2022-10-25 23:12:00.149386] --log-level=lib.eal:6 [2022-10-25 23:12:00.149393] --log-level=lib.cryptodev:5 [2022-10-25 23:12:00.149401] --log-level=user1:6 [2022-10-25 23:12:00.149409] --iova-mode=pa [2022-10-25 23:12:00.149418] --base-virtaddr=0x200000000000 [2022-10-25 23:12:00.149426] --match-allocations [2022-10-25 23:12:00.149434] --file-prefix=spdk_pid83825 [2022-10-25 23:12:00.149442] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:12:00.287764] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:12:00.329617] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:12:00.330281] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 219908.59 IOPS 859.02 MiB/s
=============================================================
Total : 219908.59 IOPS 859.02 MiB/s
[2022-10-25 23:12:13.069798] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:12:13.069876] [ DPDK EAL parameters: [2022-10-25 23:12:13.069887] bdevperf [2022-10-25 23:12:13.069896] --no-shconf [2022-10-25 23:12:13.069905] -l 0 [2022-10-25 23:12:13.069912] --huge-unlink [2022-10-25 23:12:13.069921] --log-level=lib.eal:6 [2022-10-25 23:12:13.069929] --log-level=lib.cryptodev:5 [2022-10-25 23:12:13.069936] --log-level=user1:6 [2022-10-25 23:12:13.069945] --iova-mode=pa [2022-10-25 23:12:13.069953] --base-virtaddr=0x200000000000 [2022-10-25 23:12:13.069961] --match-allocations [2022-10-25 23:12:13.069969] --file-prefix=spdk_pid83850 [2022-10-25 23:12:13.069977] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:12:13.208316] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:12:13.249582] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:12:13.250261] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 220284.56 IOPS 860.49 MiB/s
=============================================================
Total : 220284.56 IOPS 860.49 MiB/s
[2022-10-25 23:12:26.055510] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:12:26.055579] [ DPDK EAL parameters: [2022-10-25 23:12:26.055589] bdevperf [2022-10-25 23:12:26.055598] --no-shconf [2022-10-25 23:12:26.055606] -l 0 [2022-10-25 23:12:26.055614] --huge-unlink [2022-10-25 23:12:26.055621] --log-level=lib.eal:6 [2022-10-25 23:12:26.055628] --log-level=lib.cryptodev:5 [2022-10-25 23:12:26.055636] --log-level=user1:6 [2022-10-25 23:12:26.055643] --iova-mode=pa [2022-10-25 23:12:26.055650] --base-virtaddr=0x200000000000 [2022-10-25 23:12:26.055657] --match-allocations [2022-10-25 23:12:26.055665] --file-prefix=spdk_pid83875 [2022-10-25 23:12:26.055672] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:12:26.194021] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:12:26.245587] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:12:26.246263] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 220435.91 IOPS 861.08 MiB/s
=============================================================
Total : 220435.91 IOPS 861.08 MiB/s
[2022-10-25 23:12:39.062862] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:12:39.062933] [ DPDK EAL parameters: [2022-10-25 23:12:39.062944] bdevperf [2022-10-25 23:12:39.062952] --no-shconf [2022-10-25 23:12:39.062961] -l 0 [2022-10-25 23:12:39.062969] --huge-unlink [2022-10-25 23:12:39.062978] --log-level=lib.eal:6 [2022-10-25 23:12:39.062985] --log-level=lib.cryptodev:5 [2022-10-25 23:12:39.062992] --log-level=user1:6 [2022-10-25 23:12:39.063000] --iova-mode=pa [2022-10-25 23:12:39.063008] --base-virtaddr=0x200000000000 [2022-10-25 23:12:39.063016] --match-allocations [2022-10-25 23:12:39.063024] --file-prefix=spdk_pid83901 [2022-10-25 23:12:39.063033] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:12:39.201528] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:12:39.258472] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:12:39.259141] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 218110.54 IOPS 851.99 MiB/s
=============================================================
Total : 218110.54 IOPS 851.99 MiB/s
[2022-10-25 23:12:52.069012] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:12:52.069081] [ DPDK EAL parameters: [2022-10-25 23:12:52.069091] bdevperf [2022-10-25 23:12:52.069100] --no-shconf [2022-10-25 23:12:52.069108] -l 0 [2022-10-25 23:12:52.069116] --huge-unlink [2022-10-25 23:12:52.069124] --log-level=lib.eal:6 [2022-10-25 23:12:52.069131] --log-level=lib.cryptodev:5 [2022-10-25 23:12:52.069139] --log-level=user1:6 [2022-10-25 23:12:52.069146] --iova-mode=pa [2022-10-25 23:12:52.069153] --base-virtaddr=0x200000000000 [2022-10-25 23:12:52.069161] --match-allocations [2022-10-25 23:12:52.069168] --file-prefix=spdk_pid83926 [2022-10-25 23:12:52.069176] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:12:52.207851] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:12:52.253574] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:12:52.254257] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 217774.07 IOPS 850.68 MiB/s
=============================================================
Total : 217774.07 IOPS 850.68 MiB/s
[2022-10-25 23:13:05.092769] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:13:05.092840] [ DPDK EAL parameters: [2022-10-25 23:13:05.092850] bdevperf [2022-10-25 23:13:05.092859] --no-shconf [2022-10-25 23:13:05.092868] -l 0 [2022-10-25 23:13:05.092876] --huge-unlink [2022-10-25 23:13:05.092885] --log-level=lib.eal:6 [2022-10-25 23:13:05.092894] --log-level=lib.cryptodev:5 [2022-10-25 23:13:05.092901] --log-level=user1:6 [2022-10-25 23:13:05.092909] --iova-mode=pa [2022-10-25 23:13:05.092918] --base-virtaddr=0x200000000000 [2022-10-25 23:13:05.092926] --match-allocations [2022-10-25 23:13:05.092934] --file-prefix=spdk_pid83952 [2022-10-25 23:13:05.092943] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:13:05.231410] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:13:05.285594] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:13:05.286255] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 217800.45 IOPS 850.78 MiB/s
=============================================================
Total : 217800.45 IOPS 850.78 MiB/s
[2022-10-25 23:13:18.110405] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:13:18.110473] [ DPDK EAL parameters: [2022-10-25 23:13:18.110484] bdevperf [2022-10-25 23:13:18.110492] --no-shconf [2022-10-25 23:13:18.110500] -l 0 [2022-10-25 23:13:18.110508] --huge-unlink [2022-10-25 23:13:18.110515] --log-level=lib.eal:6 [2022-10-25 23:13:18.110523] --log-level=lib.cryptodev:5 [2022-10-25 23:13:18.110530] --log-level=user1:6 [2022-10-25 23:13:18.110538] --iova-mode=pa [2022-10-25 23:13:18.110544] --base-virtaddr=0x200000000000 [2022-10-25 23:13:18.110552] --match-allocations [2022-10-25 23:13:18.110559] --file-prefix=spdk_pid83977 [2022-10-25 23:13:18.110566] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:13:18.249238] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:13:18.293492] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:13:18.294175] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 0.06 IOPS 0.00 MiB/s
=============================================================
Total : 0.06 IOPS 0.00 MiB/s
[2022-10-25 23:13:31.240110] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:13:31.240182] [ DPDK EAL parameters: [2022-10-25 23:13:31.240193] bdevperf [2022-10-25 23:13:31.240202] --no-shconf [2022-10-25 23:13:31.240210] -l 0 [2022-10-25 23:13:31.240219] --huge-unlink [2022-10-25 23:13:31.240227] --log-level=lib.eal:6 [2022-10-25 23:13:31.240234] --log-level=lib.cryptodev:5 [2022-10-25 23:13:31.240241] --log-level=user1:6 [2022-10-25 23:13:31.240249] --iova-mode=pa [2022-10-25 23:13:31.240257] --base-virtaddr=0x200000000000 [2022-10-25 23:13:31.240265] --match-allocations [2022-10-25 23:13:31.240272] --file-prefix=spdk_pid84002 [2022-10-25 23:13:31.240279] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:13:31.378840] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:13:31.433486] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:13:31.434166] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 222460.54 IOPS 868.99 MiB/s
=============================================================
Total : 222460.54 IOPS 868.99 MiB/s
[2022-10-25 23:20:11.844287] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:20:11.844351] [ DPDK EAL parameters: [2022-10-25 23:20:11.844362] bdevperf [2022-10-25 23:20:11.844371] --no-shconf [2022-10-25 23:20:11.844380] -l 0 [2022-10-25 23:20:11.844388] --huge-unlink [2022-10-25 23:20:11.844396] --log-level=lib.eal:6 [2022-10-25 23:20:11.844405] --log-level=lib.cryptodev:5 [2022-10-25 23:20:11.844412] --log-level=user1:6 [2022-10-25 23:20:11.844438] --iova-mode=pa [2022-10-25 23:20:11.844448] --base-virtaddr=0x200000000000 [2022-10-25 23:20:11.844456] --match-allocations [2022-10-25 23:20:11.844465] --file-prefix=spdk_pid84034 [2022-10-25 23:20:11.844481] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:20:11.982803] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:20:12.037539] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:20:12.038213] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 222470.91 IOPS 869.03 MiB/s
=============================================================
Total : 222470.91 IOPS 869.03 MiB/s
[2022-10-25 23:20:24.779813] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:20:24.779881] [ DPDK EAL parameters: [2022-10-25 23:20:24.779892] bdevperf [2022-10-25 23:20:24.779901] --no-shconf [2022-10-25 23:20:24.779910] -l 0 [2022-10-25 23:20:24.779918] --huge-unlink [2022-10-25 23:20:24.779927] --log-level=lib.eal:6 [2022-10-25 23:20:24.779934] --log-level=lib.cryptodev:5 [2022-10-25 23:20:24.779942] --log-level=user1:6 [2022-10-25 23:20:24.779950] --iova-mode=pa [2022-10-25 23:20:24.779958] --base-virtaddr=0x200000000000 [2022-10-25 23:20:24.779966] --match-allocations [2022-10-25 23:20:24.779974] --file-prefix=spdk_pid84061 [2022-10-25 23:20:24.779983] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:20:24.918965] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:20:24.961539] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:20:24.962206] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 223754.18 IOPS 874.04 MiB/s
=============================================================
Total : 223754.18 IOPS 874.04 MiB/s
[2022-10-25 23:20:37.720662] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:20:37.720732] [ DPDK EAL parameters: [2022-10-25 23:20:37.720742] bdevperf [2022-10-25 23:20:37.720751] --no-shconf [2022-10-25 23:20:37.720760] -l 0 [2022-10-25 23:20:37.720768] --huge-unlink [2022-10-25 23:20:37.720776] --log-level=lib.eal:6 [2022-10-25 23:20:37.720784] --log-level=lib.cryptodev:5 [2022-10-25 23:20:37.720792] --log-level=user1:6 [2022-10-25 23:20:37.720800] --iova-mode=pa [2022-10-25 23:20:37.720809] --base-virtaddr=0x200000000000 [2022-10-25 23:20:37.720817] --match-allocations [2022-10-25 23:20:37.720825] --file-prefix=spdk_pid84086 [2022-10-25 23:20:37.720833] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:20:37.859472] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:20:37.909520] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:20:37.910195] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
Steps uses worklets. A description of the worklet used by
the current step is provided below. The description is extracted from the docstring
of the worklet and provided here in verbatim / as-is.
Run fio...
{'description': '\n Run fio...\n',
'extras': {'runlog': {'cmd_01': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_01.output'),
'state': {'begin': 1666725901.2728775,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7901942729949951,
'end': 1666725902.0630717,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_01.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_01.state')},
'cmd_02': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_02.output'),
'state': {'begin': 1666725902.0654507,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt"',
'cwd': 'None',
'elapsed': 11.068115472793579,
'end': 1666725913.1335661,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_02.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_02.state')},
'cmd_03': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=io_uring_cmd, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733002,\n'
' "timestamp_ms" : '
'1666733002638,\n'
' "time" : "Tue Oct 25 23:23:22 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1880924160,\n'
' "io_kbytes" : 1836840,\n'
' "bw_bytes" : 268665070,\n'
' "bw" : 262368,\n'
' "iops" : 65591.915441,\n'
' "runtime" : 7001,\n'
' "total_ios" : 459209,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2184,\n'
' "max" : 28192,\n'
' "mean" : 2371.134109,\n'
' "stddev" : 209.422890,\n'
' "N" : 459210\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 241,\n'
' "max" : 1211963,\n'
' "mean" : 12255.930946,\n'
' "stddev" : '
'2652.769239,\n'
' "N" : 459209,\n'
' "percentile" : {\n'
' "1.000000" : 11712,\n'
' "5.000000" : 11712,\n'
' "10.000000" : 11840,\n'
' "20.000000" : 11840,\n'
' "30.000000" : 11840,\n'
' "40.000000" : 11968,\n'
' "50.000000" : 11968,\n'
' "60.000000" : 12096,\n'
' "70.000000" : 12096,\n'
' "80.000000" : 12096,\n'
' "90.000000" : 12224,\n'
' "95.000000" : 12352,\n'
' "99.000000" : 21888,\n'
' "99.500000" : 23424,\n'
' "99.900000" : 31872,\n'
' "99.950000" : 50432,\n'
' "99.990000" : 64256\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11632,\n'
' "max" : 1214328,\n'
' "mean" : 14627.064896,\n'
' "stddev" : '
'2660.860079,\n'
' "N" : 459209\n'
' },\n'
' "bw_min" : 260360,\n'
' "bw_max" : 263680,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'262437.714286,\n'
' "bw_dev" : 778.330702,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 65090,\n'
' "iops_max" : 65920,\n'
' "iops_mean" : '
'65609.428571,\n'
' "iops_stddev" : '
'194.582675,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 9.857143,\n'
' "sys_cpu" : 36.157143,\n'
' "ctx" : 459183,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 98.399857,\n'
' "50" : 1.534595,\n'
' "100" : 0.054877,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=84392: Tue Oct 25 23:23:22 '
'2022\n'
' read: IOPS=65.6k, BW=256MiB/s '
'(269MB/s)(1794MiB/7001msec)\n'
' slat (nsec): min=2184, '
'max=28192, avg=2371.13, '
'stdev=209.42\n'
' clat (nsec): min=241, '
'max=1212.0k, avg=12255.93, '
'stdev=2652.77\n'
' lat (usec): min=11, '
'max=1214, avg=14.63, stdev= 2.66\n'
' clat percentiles (nsec):\n'
' | 1.00th=[11712], '
'5.00th=[11712], 10.00th=[11840], '
'20.00th=[11840],\n'
' | 30.00th=[11840], '
'40.00th=[11968], 50.00th=[11968], '
'60.00th=[12096],\n'
' | 70.00th=[12096], '
'80.00th=[12096], 90.00th=[12224], '
'95.00th=[12352],\n'
' | 99.00th=[21888], '
'99.50th=[23424], 99.90th=[31872], '
'99.95th=[50432],\n'
' | 99.99th=[64256]\n'
' bw ( KiB/s): min=260360, '
'max=263680, per=100.00%, '
'avg=262437.71, stdev=778.33, '
'samples=14\n'
' iops : min=65090, '
'max=65920, avg=65609.43, '
'stdev=194.58, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%, 750=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=98.40%, 50=1.53%, '
'100=0.05%\n'
' lat (usec) : 250=0.01%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=9.86%, '
'sys=36.16%, ctx=459183, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=459209,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=256MiB/s (269MB/s), '
'256MiB/s-256MiB/s '
'(269MB/s-269MB/s), io=1794MiB '
'(1881MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_03.output'),
'state': {'begin': 1666725913.1364834,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7891125679016113,
'end': 1666725913.925596,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_03.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_03.state')},
'cmd_04': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_04.output'),
'state': {'begin': 1666725914.9139788,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7808358669281006,
'end': 1666725915.6948147,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_04.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_04.state')},
'cmd_05': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_05.output'),
'state': {'begin': 1666725915.696932,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt"',
'cwd': 'None',
'elapsed': 11.044736385345459,
'end': 1666725926.7416685,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_05.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_05.state')},
'cmd_06': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=io_uring_cmd, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733016,\n'
' "timestamp_ms" : '
'1666733016245,\n'
' "time" : "Tue Oct 25 23:23:36 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1887145984,\n'
' "io_kbytes" : 1842916,\n'
' "bw_bytes" : 269553775,\n'
' "bw" : 263236,\n'
' "iops" : 65809.027282,\n'
' "runtime" : 7001,\n'
' "total_ios" : 460729,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2174,\n'
' "max" : 31970,\n'
' "mean" : 2353.754324,\n'
' "stddev" : 207.012099,\n'
' "N" : 460729\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 200,\n'
' "max" : 183274,\n'
' "mean" : 12205.161577,\n'
' "stddev" : '
'1978.440122,\n'
' "N" : 460729,\n'
' "percentile" : {\n'
' "1.000000" : 11712,\n'
' "5.000000" : 11712,\n'
' "10.000000" : 11712,\n'
' "20.000000" : 11840,\n'
' "30.000000" : 11840,\n'
' "40.000000" : 11840,\n'
' "50.000000" : 11968,\n'
' "60.000000" : 11968,\n'
' "70.000000" : 12096,\n'
' "80.000000" : 12096,\n'
' "90.000000" : 12096,\n'
' "95.000000" : 12352,\n'
' "99.000000" : 21888,\n'
' "99.500000" : 23168,\n'
' "99.900000" : 31360,\n'
' "99.950000" : 49920,\n'
' "99.990000" : 64256\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11591,\n'
' "max" : 185708,\n'
' "mean" : 14558.915901,\n'
' "stddev" : '
'1988.143930,\n'
' "N" : 460729\n'
' },\n'
' "bw_min" : 262208,\n'
' "bw_max" : 263920,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'263299.928571,\n'
' "bw_dev" : 429.320118,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 65552,\n'
' "iops_max" : 65980,\n'
' "iops_mean" : '
'65824.928571,\n'
' "iops_stddev" : '
'107.268509,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 9.728571,\n'
' "sys_cpu" : 36.385714,\n'
' "ctx" : 460697,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 98.392330,\n'
' "50" : 1.542989,\n'
' "100" : 0.051874,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=84430: Tue Oct 25 23:23:36 '
'2022\n'
' read: IOPS=65.8k, BW=257MiB/s '
'(270MB/s)(1800MiB/7001msec)\n'
' slat (nsec): min=2174, '
'max=31970, avg=2353.75, '
'stdev=207.01\n'
' clat (nsec): min=200, '
'max=183274, avg=12205.16, '
'stdev=1978.44\n'
' lat (usec): min=11, max=185, '
'avg=14.56, stdev= 1.99\n'
' clat percentiles (nsec):\n'
' | 1.00th=[11712], '
'5.00th=[11712], 10.00th=[11712], '
'20.00th=[11840],\n'
' | 30.00th=[11840], '
'40.00th=[11840], 50.00th=[11968], '
'60.00th=[11968],\n'
' | 70.00th=[12096], '
'80.00th=[12096], 90.00th=[12096], '
'95.00th=[12352],\n'
' | 99.00th=[21888], '
'99.50th=[23168], 99.90th=[31360], '
'99.95th=[49920],\n'
' | 99.99th=[64256]\n'
' bw ( KiB/s): min=262208, '
'max=263920, per=100.00%, '
'avg=263299.93, stdev=429.32, '
'samples=14\n'
' iops : min=65552, '
'max=65980, avg=65824.93, '
'stdev=107.27, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 10=0.01%, '
'20=98.39%, 50=1.54%, 100=0.05%, '
'250=0.01%\n'
' cpu : usr=9.73%, '
'sys=36.39%, ctx=460697, majf=0, '
'minf=1\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=460729,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=257MiB/s (270MB/s), '
'257MiB/s-257MiB/s '
'(270MB/s-270MB/s), io=1800MiB '
'(1887MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_06.output'),
'state': {'begin': 1666725926.7443569,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7978301048278809,
'end': 1666725927.542187,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_06.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_06.state')},
'cmd_07': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_07.output'),
'state': {'begin': 1666725928.5928621,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7860045433044434,
'end': 1666725929.3788667,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_07.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_07.state')},
'cmd_08': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_08.output'),
'state': {'begin': 1666725929.3810534,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt"',
'cwd': 'None',
'elapsed': 11.049619674682617,
'end': 1666725940.4306731,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_08.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_08.state')},
'cmd_09': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=io_uring_cmd, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733029,\n'
' "timestamp_ms" : '
'1666733029933,\n'
' "time" : "Tue Oct 25 23:23:49 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1777991680,\n'
' "io_kbytes" : 1736320,\n'
' "bw_bytes" : 253962531,\n'
' "bw" : 248010,\n'
' "iops" : 62002.571061,\n'
' "runtime" : 7001,\n'
' "total_ios" : 434080,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2184,\n'
' "max" : 31269,\n'
' "mean" : 2476.576864,\n'
' "stddev" : 207.606246,\n'
' "N" : 434080\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 250,\n'
' "max" : 133119,\n'
' "mean" : 13071.487523,\n'
' "stddev" : '
'2165.425154,\n'
' "N" : 434080,\n'
' "percentile" : {\n'
' "1.000000" : 11968,\n'
' "5.000000" : 11968,\n'
' "10.000000" : 12096,\n'
' "20.000000" : 12096,\n'
' "30.000000" : 12096,\n'
' "40.000000" : 12224,\n'
' "50.000000" : 13248,\n'
' "60.000000" : 13376,\n'
' "70.000000" : 13376,\n'
' "80.000000" : 13376,\n'
' "90.000000" : 13504,\n'
' "95.000000" : 13632,\n'
' "99.000000" : 22912,\n'
' "99.500000" : 29312,\n'
' "99.900000" : 32640,\n'
' "99.950000" : 50432,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 13816,\n'
' "max" : 135424,\n'
' "mean" : 15548.064387,\n'
' "stddev" : '
'2206.175957,\n'
' "N" : 434080\n'
' },\n'
' "bw_min" : 237664,\n'
' "bw_max" : 261000,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'248095.357143,\n'
' "bw_dev" : 11422.448246,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 59416,\n'
' "iops_max" : 65250,\n'
' "iops_mean" : '
'62023.785714,\n'
' "iops_stddev" : '
'2855.663204,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 8.014286,\n'
' "sys_cpu" : 35.057143,\n'
' "ctx" : 434068,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 98.284187,\n'
' "50" : 1.652691,\n'
' "100" : 0.052525,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=84468: Tue Oct 25 23:23:49 '
'2022\n'
' read: IOPS=62.0k, BW=242MiB/s '
'(254MB/s)(1696MiB/7001msec)\n'
' slat (nsec): min=2184, '
'max=31269, avg=2476.58, '
'stdev=207.61\n'
' clat (nsec): min=250, '
'max=133119, avg=13071.49, '
'stdev=2165.43\n'
' lat (usec): min=13, max=135, '
'avg=15.55, stdev= 2.21\n'
' clat percentiles (nsec):\n'
' | 1.00th=[11968], '
'5.00th=[11968], 10.00th=[12096], '
'20.00th=[12096],\n'
' | 30.00th=[12096], '
'40.00th=[12224], 50.00th=[13248], '
'60.00th=[13376],\n'
' | 70.00th=[13376], '
'80.00th=[13376], 90.00th=[13504], '
'95.00th=[13632],\n'
' | 99.00th=[22912], '
'99.50th=[29312], 99.90th=[32640], '
'99.95th=[50432],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=237664, '
'max=261000, per=100.00%, '
'avg=248095.36, stdev=11422.45, '
'samples=14\n'
' iops : min=59416, '
'max=65250, avg=62023.79, '
'stdev=2855.66, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 10=0.01%, '
'20=98.28%, 50=1.65%, 100=0.05%, '
'250=0.01%\n'
' cpu : usr=8.01%, '
'sys=35.06%, ctx=434068, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=434080,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=242MiB/s (254MB/s), '
'242MiB/s-242MiB/s '
'(254MB/s-254MB/s), io=1696MiB '
'(1778MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_09.output'),
'state': {'begin': 1666725940.4336092,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7855269908905029,
'end': 1666725941.2191362,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_09.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_09.state')},
'cmd_10': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_10.output'),
'state': {'begin': 1666725942.249782,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7795379161834717,
'end': 1666725943.02932,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_10.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_10.state')},
'cmd_100': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_100.output'),
'state': {'begin': 1666726373.990537,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7794060707092285,
'end': 1666726374.769943,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_100.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_100.state')},
'cmd_101': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_101.output'),
'state': {'begin': 1666726374.7717996,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt"',
'cwd': 'None',
'elapsed': 11.049149751663208,
'end': 1666726385.8209493,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_101.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_101.state')},
'cmd_102': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, '
'ioengine=io_uring_cmd, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733475,\n'
' "timestamp_ms" : '
'1666733475316,\n'
' "time" : "Tue Oct 25 23:31:15 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2836426752,\n'
' "io_kbytes" : 2769948,\n'
' "bw_bytes" : 405145943,\n'
' "bw" : 395650,\n'
' "iops" : 98912.298243,\n'
' "runtime" : 7001,\n'
' "total_ios" : 692485,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1893,\n'
' "max" : 27962,\n'
' "mean" : 2200.003763,\n'
' "stddev" : '
'323.295581,\n'
' "N" : 692486\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 3106,\n'
' "max" : 189856,\n'
' "mean" : '
'17485.482183,\n'
' "stddev" : '
'3150.118639,\n'
' "N" : 692486,\n'
' "percentile" : {\n'
' "1.000000" : 16320,\n'
' "5.000000" : 16512,\n'
' "10.000000" : '
'16512,\n'
' "20.000000" : '
'16512,\n'
' "30.000000" : '
'16512,\n'
' "40.000000" : '
'16768,\n'
' "50.000000" : '
'17024,\n'
' "60.000000" : '
'17024,\n'
' "70.000000" : '
'17024,\n'
' "80.000000" : '
'17280,\n'
' "90.000000" : '
'17280,\n'
' "95.000000" : '
'19584,\n'
' "99.000000" : '
'33536,\n'
' "99.500000" : '
'34560,\n'
' "99.900000" : '
'50432,\n'
' "99.950000" : '
'62720,\n'
' "99.990000" : 73216\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11261,\n'
' "max" : 191810,\n'
' "mean" : '
'19685.485946,\n'
' "stddev" : '
'3142.083143,\n'
' "N" : 692486\n'
' },\n'
' "bw_min" : 394512,\n'
' "bw_max" : 396576,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'395746.285714,\n'
' "bw_dev" : 618.185363,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 98628,\n'
' "iops_max" : 99144,\n'
' "iops_mean" : '
'98936.571429,\n'
' "iops_stddev" : '
'154.546341,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 12.085714,\n'
' "sys_cpu" : 38.200000,\n'
' "ctx" : 346121,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010686,\n'
' "20" : 95.026318,\n'
' "50" : 4.855268,\n'
' "100" : 0.104840,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=85837: Tue Oct 25 23:31:15 '
'2022\n'
' read: IOPS=98.9k, BW=386MiB/s '
'(405MB/s)(2705MiB/7001msec)\n'
' slat (nsec): min=1893, '
'max=27962, avg=2200.00, '
'stdev=323.30\n'
' clat (usec): min=3, max=189, '
'avg=17.49, stdev= 3.15\n'
' lat (usec): min=11, '
'max=191, avg=19.69, stdev= 3.14\n'
' clat percentiles (nsec):\n'
' | 1.00th=[16320], '
'5.00th=[16512], 10.00th=[16512], '
'20.00th=[16512],\n'
' | 30.00th=[16512], '
'40.00th=[16768], '
'50.00th=[17024], '
'60.00th=[17024],\n'
' | 70.00th=[17024], '
'80.00th=[17280], '
'90.00th=[17280], '
'95.00th=[19584],\n'
' | 99.00th=[33536], '
'99.50th=[34560], '
'99.90th=[50432], '
'99.95th=[62720],\n'
' | 99.99th=[73216]\n'
' bw ( KiB/s): min=394512, '
'max=396576, per=100.00%, '
'avg=395746.29, stdev=618.19, '
'samples=14\n'
' iops : min=98628, '
'max=99144, avg=98936.57, '
'stdev=154.55, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=95.03%, 50=4.86%, '
'100=0.10%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=12.09%, '
'sys=38.20%, ctx=346121, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=692485,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=386MiB/s (405MB/s), '
'386MiB/s-386MiB/s '
'(405MB/s-405MB/s), io=2705MiB '
'(2836MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_102.output'),
'state': {'begin': 1666726385.8235474,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7883989810943604,
'end': 1666726386.6119463,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_102.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_102.state')},
'cmd_103': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_103.output'),
'state': {'begin': 1666726387.6444223,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7858450412750244,
'end': 1666726388.4302673,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_103.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_103.state')},
'cmd_104': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_104.output'),
'state': {'begin': 1666726388.4321644,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt"',
'cwd': 'None',
'elapsed': 11.038004875183105,
'end': 1666726399.4701693,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_104.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_104.state')},
'cmd_105': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, '
'ioengine=io_uring_cmd, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733488,\n'
' "timestamp_ms" : '
'1666733488966,\n'
' "time" : "Tue Oct 25 23:31:28 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2790170624,\n'
' "io_kbytes" : 2724776,\n'
' "bw_bytes" : 398538869,\n'
' "bw" : 389198,\n'
' "iops" : 97299.242965,\n'
' "runtime" : 7001,\n'
' "total_ios" : 681192,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1963,\n'
' "max" : 37029,\n'
' "mean" : 2279.325263,\n'
' "stddev" : '
'339.765872,\n'
' "N" : 681193\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 3075,\n'
' "max" : 121388,\n'
' "mean" : '
'17720.137531,\n'
' "stddev" : '
'3211.933311,\n'
' "N" : 681193,\n'
' "percentile" : {\n'
' "1.000000" : 16512,\n'
' "5.000000" : 16512,\n'
' "10.000000" : '
'16768,\n'
' "20.000000" : '
'16768,\n'
' "30.000000" : '
'16768,\n'
' "40.000000" : '
'16768,\n'
' "50.000000" : '
'17280,\n'
' "60.000000" : '
'17280,\n'
' "70.000000" : '
'17280,\n'
' "80.000000" : '
'17536,\n'
' "90.000000" : '
'17536,\n'
' "95.000000" : '
'20096,\n'
' "99.000000" : '
'34048,\n'
' "99.500000" : '
'35072,\n'
' "99.900000" : '
'50432,\n'
' "99.950000" : '
'63232,\n'
' "99.990000" : 74240\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11381,\n'
' "max" : 123462,\n'
' "mean" : '
'19999.462794,\n'
' "stddev" : '
'3203.796715,\n'
' "N" : 681193\n'
' },\n'
' "bw_min" : 388424,\n'
' "bw_max" : 390448,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'389292.714286,\n'
' "bw_dev" : 718.143593,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 97106,\n'
' "iops_max" : 97612,\n'
' "iops_mean" : '
'97323.142857,\n'
' "iops_stddev" : '
'179.542520,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 11.500000,\n'
' "sys_cpu" : 39.285714,\n'
' "ctx" : 340448,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.011304,\n'
' "20" : 94.985261,\n'
' "50" : 4.893921,\n'
' "100" : 0.104082,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=85875: Tue Oct 25 23:31:28 '
'2022\n'
' read: IOPS=97.3k, BW=380MiB/s '
'(399MB/s)(2661MiB/7001msec)\n'
' slat (nsec): min=1963, '
'max=37029, avg=2279.33, '
'stdev=339.77\n'
' clat (usec): min=3, max=121, '
'avg=17.72, stdev= 3.21\n'
' lat (usec): min=11, '
'max=123, avg=20.00, stdev= 3.20\n'
' clat percentiles (nsec):\n'
' | 1.00th=[16512], '
'5.00th=[16512], 10.00th=[16768], '
'20.00th=[16768],\n'
' | 30.00th=[16768], '
'40.00th=[16768], '
'50.00th=[17280], '
'60.00th=[17280],\n'
' | 70.00th=[17280], '
'80.00th=[17536], '
'90.00th=[17536], '
'95.00th=[20096],\n'
' | 99.00th=[34048], '
'99.50th=[35072], '
'99.90th=[50432], '
'99.95th=[63232],\n'
' | 99.99th=[74240]\n'
' bw ( KiB/s): min=388424, '
'max=390448, per=100.00%, '
'avg=389292.71, stdev=718.14, '
'samples=14\n'
' iops : min=97106, '
'max=97612, avg=97323.14, '
'stdev=179.54, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=94.99%, 50=4.89%, '
'100=0.10%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=11.50%, '
'sys=39.29%, ctx=340448, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=681192,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=380MiB/s (399MB/s), '
'380MiB/s-380MiB/s '
'(399MB/s-399MB/s), io=2661MiB '
'(2790MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_105.output'),
'state': {'begin': 1666726399.4770014,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7884995937347412,
'end': 1666726400.265501,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_105.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_105.state')},
'cmd_106': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_106.output'),
'state': {'begin': 1666726401.2915058,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7873985767364502,
'end': 1666726402.0789044,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_106.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_106.state')},
'cmd_107': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_107.output'),
'state': {'begin': 1666726402.0813653,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt"',
'cwd': 'None',
'elapsed': 11.045714855194092,
'end': 1666726413.1270802,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_107.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_107.state')},
'cmd_108': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, '
'ioengine=io_uring_cmd, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733502,\n'
' "timestamp_ms" : '
'1666733502622,\n'
' "time" : "Tue Oct 25 23:31:42 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2812887040,\n'
' "io_kbytes" : 2746960,\n'
' "bw_bytes" : 401783608,\n'
' "bw" : 392366,\n'
' "iops" : 98091.558349,\n'
' "runtime" : 7001,\n'
' "total_ios" : 686739,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1884,\n'
' "max" : 31248,\n'
' "mean" : 2222.947482,\n'
' "stddev" : '
'326.104416,\n'
' "N" : 686739\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2996,\n'
' "max" : 255970,\n'
' "mean" : '
'17635.028845,\n'
' "stddev" : '
'3256.648995,\n'
' "N" : 686740,\n'
' "percentile" : {\n'
' "1.000000" : 16512,\n'
' "5.000000" : 16512,\n'
' "10.000000" : '
'16512,\n'
' "20.000000" : '
'16768,\n'
' "30.000000" : '
'16768,\n'
' "40.000000" : '
'16768,\n'
' "50.000000" : '
'17024,\n'
' "60.000000" : '
'17280,\n'
' "70.000000" : '
'17280,\n'
' "80.000000" : '
'17280,\n'
' "90.000000" : '
'17536,\n'
' "95.000000" : '
'20096,\n'
' "99.000000" : '
'34048,\n'
' "99.500000" : '
'34560,\n'
' "99.900000" : '
'50944,\n'
' "99.950000" : '
'63744,\n'
' "99.990000" : 74240\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11231,\n'
' "max" : 257904,\n'
' "mean" : '
'19857.976051,\n'
' "stddev" : '
'3249.803901,\n'
' "N" : 686740\n'
' },\n'
' "bw_min" : 388705,\n'
' "bw_max" : 395896,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'392460.642857,\n'
' "bw_dev" : 2587.148404,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 97176,\n'
' "iops_max" : 98974,\n'
' "iops_mean" : '
'98115.142857,\n'
' "iops_stddev" : '
'646.815020,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 12.271429,\n'
' "sys_cpu" : 38.271429,\n'
' "ctx" : 343225,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.014416,\n'
' "20" : 94.981354,\n'
' "50" : 4.886718,\n'
' "100" : 0.113435,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=85913: Tue Oct 25 23:31:42 '
'2022\n'
' read: IOPS=98.1k, BW=383MiB/s '
'(402MB/s)(2683MiB/7001msec)\n'
' slat (nsec): min=1884, '
'max=31248, avg=2222.95, '
'stdev=326.10\n'
' clat (usec): min=2, max=255, '
'avg=17.64, stdev= 3.26\n'
' lat (usec): min=11, '
'max=257, avg=19.86, stdev= 3.25\n'
' clat percentiles (nsec):\n'
' | 1.00th=[16512], '
'5.00th=[16512], 10.00th=[16512], '
'20.00th=[16768],\n'
' | 30.00th=[16768], '
'40.00th=[16768], '
'50.00th=[17024], '
'60.00th=[17280],\n'
' | 70.00th=[17280], '
'80.00th=[17280], '
'90.00th=[17536], '
'95.00th=[20096],\n'
' | 99.00th=[34048], '
'99.50th=[34560], '
'99.90th=[50944], '
'99.95th=[63744],\n'
' | 99.99th=[74240]\n'
' bw ( KiB/s): min=388705, '
'max=395896, per=100.00%, '
'avg=392460.64, stdev=2587.15, '
'samples=14\n'
' iops : min=97176, '
'max=98974, avg=98115.14, '
'stdev=646.82, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=94.98%, 50=4.89%, '
'100=0.11%\n'
' lat (usec) : 250=0.01%, '
'500=0.01%\n'
' cpu : usr=12.27%, '
'sys=38.27%, ctx=343225, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=686739,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=383MiB/s (402MB/s), '
'383MiB/s-383MiB/s '
'(402MB/s-402MB/s), io=2683MiB '
'(2813MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_108.output'),
'state': {'begin': 1666726413.1298363,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.788353443145752,
'end': 1666726413.9181898,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_108.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_108.state')},
'cmd_109': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_109.output'),
'state': {'begin': 1666726414.953366,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7746560573577881,
'end': 1666726415.728022,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_109.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_109.state')},
'cmd_11': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_11.output'),
'state': {'begin': 1666725943.0313234,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.058010816574097,
'end': 1666725954.0893342,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_11.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_11.state')},
'cmd_110': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_110.output'),
'state': {'begin': 1666726415.7300074,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.052646398544312,
'end': 1666726426.7826538,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_110.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_110.state')},
'cmd_111': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733516,\n'
' "timestamp_ms" : '
'1666733516277,\n'
' "time" : "Tue Oct 25 23:31:56 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2816339968,\n'
' "io_kbytes" : 2750332,\n'
' "bw_bytes" : 402276813,\n'
' "bw" : 392848,\n'
' "iops" : 98211.969719,\n'
' "runtime" : 7001,\n'
' "total_ios" : 687582,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2003,\n'
' "max" : 40265,\n'
' "mean" : 2351.189732,\n'
' "stddev" : '
'337.786924,\n'
' "N" : 687582\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 3356,\n'
' "max" : 209222,\n'
' "mean" : '
'17502.922175,\n'
' "stddev" : '
'3232.779953,\n'
' "N" : 687583,\n'
' "percentile" : {\n'
' "1.000000" : 16320,\n'
' "5.000000" : 16512,\n'
' "10.000000" : '
'16512,\n'
' "20.000000" : '
'16512,\n'
' "30.000000" : '
'16512,\n'
' "40.000000" : '
'16768,\n'
' "50.000000" : '
'17024,\n'
' "60.000000" : '
'17024,\n'
' "70.000000" : '
'17280,\n'
' "80.000000" : '
'17280,\n'
' "90.000000" : '
'17280,\n'
' "95.000000" : '
'19584,\n'
' "99.000000" : '
'33536,\n'
' "99.500000" : '
'34560,\n'
' "99.900000" : '
'50432,\n'
' "99.950000" : '
'63744,\n'
' "99.990000" : 74240\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11251,\n'
' "max" : 211306,\n'
' "mean" : '
'19854.111460,\n'
' "stddev" : '
'3223.695868,\n'
' "N" : 687583\n'
' },\n'
' "bw_min" : 391559,\n'
' "bw_max" : 394072,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'392940.500000,\n'
' "bw_dev" : 744.587315,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 97889,\n'
' "iops_max" : 98520,\n'
' "iops_mean" : '
'98235.214286,\n'
' "iops_stddev" : '
'186.488267,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 12.885714,\n'
' "sys_cpu" : 39.400000,\n'
' "ctx" : 343653,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010471,\n'
' "20" : 95.019649,\n'
' "50" : 4.858766,\n'
' "100" : 0.108642,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=85953: Tue '
'Oct 25 23:31:56 2022\n'
' read: IOPS=98.2k, BW=384MiB/s '
'(402MB/s)(2686MiB/7001msec)\n'
' slat (nsec): min=2003, '
'max=40265, avg=2351.19, '
'stdev=337.79\n'
' clat (usec): min=3, max=209, '
'avg=17.50, stdev= 3.23\n'
' lat (usec): min=11, '
'max=211, avg=19.85, stdev= 3.22\n'
' clat percentiles (nsec):\n'
' | 1.00th=[16320], '
'5.00th=[16512], 10.00th=[16512], '
'20.00th=[16512],\n'
' | 30.00th=[16512], '
'40.00th=[16768], '
'50.00th=[17024], '
'60.00th=[17024],\n'
' | 70.00th=[17280], '
'80.00th=[17280], '
'90.00th=[17280], '
'95.00th=[19584],\n'
' | 99.00th=[33536], '
'99.50th=[34560], '
'99.90th=[50432], '
'99.95th=[63744],\n'
' | 99.99th=[74240]\n'
' bw ( KiB/s): min=391559, '
'max=394072, per=100.00%, '
'avg=392940.50, stdev=744.59, '
'samples=14\n'
' iops : min=97889, '
'max=98520, avg=98235.21, '
'stdev=186.49, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=95.02%, 50=4.86%, '
'100=0.11%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=12.89%, '
'sys=39.40%, ctx=343653, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=687582,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=384MiB/s (402MB/s), '
'384MiB/s-384MiB/s '
'(402MB/s-402MB/s), io=2686MiB '
'(2816MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_111.output'),
'state': {'begin': 1666726426.7856476,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7886650562286377,
'end': 1666726427.5743127,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_111.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_111.state')},
'cmd_112': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_112.output'),
'state': {'begin': 1666726428.5737777,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7817878723144531,
'end': 1666726429.3555655,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_112.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_112.state')},
'cmd_113': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_113.output'),
'state': {'begin': 1666726429.35767,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.057876348495483,
'end': 1666726440.4155464,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_113.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_113.state')},
'cmd_114': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733529,\n'
' "timestamp_ms" : '
'1666733529910,\n'
' "time" : "Tue Oct 25 23:32:09 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2630627328,\n'
' "io_kbytes" : 2568972,\n'
' "bw_bytes" : 375750225,\n'
' "bw" : 366943,\n'
' "iops" : 91735.752035,\n'
' "runtime" : 7001,\n'
' "total_ios" : 642242,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2124,\n'
' "max" : 14818,\n'
' "mean" : 2496.051688,\n'
' "stddev" : '
'247.656662,\n'
' "N" : 642242\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4859,\n'
' "max" : 136816,\n'
' "mean" : '
'18815.614652,\n'
' "stddev" : '
'3328.463438,\n'
' "N" : 642243,\n'
' "percentile" : {\n'
' "1.000000" : 17536,\n'
' "5.000000" : 17792,\n'
' "10.000000" : '
'17792,\n'
' "20.000000" : '
'17792,\n'
' "30.000000" : '
'17792,\n'
' "40.000000" : '
'18048,\n'
' "50.000000" : '
'18304,\n'
' "60.000000" : '
'18304,\n'
' "70.000000" : '
'18304,\n'
' "80.000000" : '
'18560,\n'
' "90.000000" : '
'18560,\n'
' "95.000000" : '
'21888,\n'
' "99.000000" : '
'35072,\n'
' "99.500000" : '
'44288,\n'
' "99.900000" : '
'49920,\n'
' "99.950000" : '
'62208,\n'
' "99.990000" : 74240\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12043,\n'
' "max" : 139101,\n'
' "mean" : '
'21311.665947,\n'
' "stddev" : '
'3318.507739,\n'
' "N" : 642243\n'
' },\n'
' "bw_min" : 366640,\n'
' "bw_max" : 368008,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'367114.357143,\n'
' "bw_dev" : 409.196136,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 91660,\n'
' "iops_max" : 92002,\n'
' "iops_mean" : '
'91778.500000,\n'
' "iops_stddev" : '
'102.304034,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 12.457143,\n'
' "sys_cpu" : 36.600000,\n'
' "ctx" : 321062,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 94.633643,\n'
' "50" : 5.261257,\n'
' "100" : 0.097627,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=85992: Tue '
'Oct 25 23:32:09 2022\n'
' read: IOPS=91.7k, BW=358MiB/s '
'(376MB/s)(2509MiB/7001msec)\n'
' slat (nsec): min=2124, '
'max=14818, avg=2496.05, '
'stdev=247.66\n'
' clat (usec): min=4, max=136, '
'avg=18.82, stdev= 3.33\n'
' lat (usec): min=12, '
'max=139, avg=21.31, stdev= 3.32\n'
' clat percentiles (nsec):\n'
' | 1.00th=[17536], '
'5.00th=[17792], 10.00th=[17792], '
'20.00th=[17792],\n'
' | 30.00th=[17792], '
'40.00th=[18048], '
'50.00th=[18304], '
'60.00th=[18304],\n'
' | 70.00th=[18304], '
'80.00th=[18560], '
'90.00th=[18560], '
'95.00th=[21888],\n'
' | 99.00th=[35072], '
'99.50th=[44288], '
'99.90th=[49920], '
'99.95th=[62208],\n'
' | 99.99th=[74240]\n'
' bw ( KiB/s): min=366640, '
'max=368008, per=100.00%, '
'avg=367114.36, stdev=409.20, '
'samples=14\n'
' iops : min=91660, '
'max=92002, avg=91778.50, '
'stdev=102.30, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=94.63%, 50=5.26%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=12.46%, '
'sys=36.60%, ctx=321062, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=642242,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=358MiB/s (376MB/s), '
'358MiB/s-358MiB/s '
'(376MB/s-376MB/s), io=2509MiB '
'(2631MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_114.output'),
'state': {'begin': 1666726440.4181273,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7908730506896973,
'end': 1666726441.2090003,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_114.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_114.state')},
'cmd_115': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_115.output'),
'state': {'begin': 1666726442.2077224,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.782257080078125,
'end': 1666726442.9899795,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_115.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_115.state')},
'cmd_116': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_116.output'),
'state': {'begin': 1666726442.992052,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.055752992630005,
'end': 1666726454.047805,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_116.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_116.state')},
'cmd_117': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733543,\n'
' "timestamp_ms" : '
'1666733543542,\n'
' "time" : "Tue Oct 25 23:32:23 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2749571072,\n'
' "io_kbytes" : 2685128,\n'
' "bw_bytes" : 392739761,\n'
' "bw" : 383534,\n'
' "iops" : 95883.588059,\n'
' "runtime" : 7001,\n'
' "total_ios" : 671281,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2064,\n'
' "max" : 31810,\n'
' "mean" : 2400.967224,\n'
' "stddev" : '
'306.105301,\n'
' "N" : 671281\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 391,\n'
' "max" : 135253,\n'
' "mean" : '
'17952.067264,\n'
' "stddev" : '
'3287.475107,\n'
' "N" : 671282,\n'
' "percentile" : {\n'
' "1.000000" : 16512,\n'
' "5.000000" : 16512,\n'
' "10.000000" : '
'16768,\n'
' "20.000000" : '
'16768,\n'
' "30.000000" : '
'16768,\n'
' "40.000000" : '
'17280,\n'
' "50.000000" : '
'17280,\n'
' "60.000000" : '
'17280,\n'
' "70.000000" : '
'17792,\n'
' "80.000000" : '
'18048,\n'
' "90.000000" : '
'18304,\n'
' "95.000000" : '
'20352,\n'
' "99.000000" : '
'34048,\n'
' "99.500000" : '
'37120,\n'
' "99.900000" : '
'50432,\n'
' "99.950000" : '
'63744,\n'
' "99.990000" : 74240\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11592,\n'
' "max" : 143409,\n'
' "mean" : '
'20353.034373,\n'
' "stddev" : '
'3289.918264,\n'
' "N" : 671282\n'
' },\n'
' "bw_min" : 367224,\n'
' "bw_max" : 390992,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'383631.928571,\n'
' "bw_dev" : 9319.619474,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 91806,\n'
' "iops_max" : 97748,\n'
' "iops_mean" : '
'95907.928571,\n'
' "iops_stddev" : '
'2330.003977,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.528571,\n'
' "sys_cpu" : 37.685714,\n'
' "ctx" : 335517,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.011471,\n'
' "20" : 94.923586,\n'
' "50" : 4.958132,\n'
' "100" : 0.104278,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=86031: Tue '
'Oct 25 23:32:23 2022\n'
' read: IOPS=95.9k, BW=375MiB/s '
'(393MB/s)(2622MiB/7001msec)\n'
' slat (nsec): min=2064, '
'max=31810, avg=2400.97, '
'stdev=306.11\n'
' clat (nsec): min=391, '
'max=135253, avg=17952.07, '
'stdev=3287.48\n'
' lat (usec): min=11, '
'max=143, avg=20.35, stdev= 3.29\n'
' clat percentiles (nsec):\n'
' | 1.00th=[16512], '
'5.00th=[16512], 10.00th=[16768], '
'20.00th=[16768],\n'
' | 30.00th=[16768], '
'40.00th=[17280], '
'50.00th=[17280], '
'60.00th=[17280],\n'
' | 70.00th=[17792], '
'80.00th=[18048], '
'90.00th=[18304], '
'95.00th=[20352],\n'
' | 99.00th=[34048], '
'99.50th=[37120], '
'99.90th=[50432], '
'99.95th=[63744],\n'
' | 99.99th=[74240]\n'
' bw ( KiB/s): min=367224, '
'max=390992, per=100.00%, '
'avg=383631.93, stdev=9319.62, '
'samples=14\n'
' iops : min=91806, '
'max=97748, avg=95907.93, '
'stdev=2330.00, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=94.92%, 50=4.96%, '
'100=0.10%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=13.53%, '
'sys=37.69%, ctx=335517, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=671281,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=375MiB/s (393MB/s), '
'375MiB/s-375MiB/s '
'(393MB/s-393MB/s), io=2622MiB '
'(2750MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_117.output'),
'state': {'begin': 1666726454.0507383,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.793297290802002,
'end': 1666726454.8440356,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_117.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_117.state')},
'cmd_118': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_118.output'),
'state': {'begin': 1666726456.863824,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7858519554138184,
'end': 1666726457.6496758,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_118.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_118.state')},
'cmd_119': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_119.output'),
'state': {'begin': 1666726457.6520848,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.47193956375122,
'end': 1666726469.1240244,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_119.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_119.state')},
'cmd_12': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733043,\n'
' "timestamp_ms" : '
'1666733043592,\n'
' "time" : "Tue Oct 25 23:24:03 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1862557696,\n'
' "io_kbytes" : 1818904,\n'
' "bw_bytes" : 266041664,\n'
' "bw" : 259806,\n'
' "iops" : 64951.578346,\n'
' "runtime" : 7001,\n'
' "total_ios" : 454726,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2314,\n'
' "max" : 40516,\n'
' "mean" : 2487.807033,\n'
' "stddev" : 208.501404,\n'
' "N" : 454726\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 351,\n'
' "max" : 1341115,\n'
' "mean" : 12323.750791,\n'
' "stddev" : '
'2811.725511,\n'
' "N" : 454726,\n'
' "percentile" : {\n'
' "1.000000" : 11840,\n'
' "5.000000" : 11840,\n'
' "10.000000" : 11840,\n'
' "20.000000" : 11840,\n'
' "30.000000" : 11968,\n'
' "40.000000" : 11968,\n'
' "50.000000" : 12096,\n'
' "60.000000" : 12096,\n'
' "70.000000" : 12096,\n'
' "80.000000" : 12224,\n'
' "90.000000" : 12224,\n'
' "95.000000" : 12480,\n'
' "99.000000" : 22144,\n'
' "99.500000" : 23680,\n'
' "99.900000" : 31616,\n'
' "99.950000" : 50432,\n'
' "99.990000" : 64256\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12694,\n'
' "max" : 1343630,\n'
' "mean" : 14811.557824,\n'
' "stddev" : '
'2820.004980,\n'
' "N" : 454726\n'
' },\n'
' "bw_min" : 259104,\n'
' "bw_max" : 260440,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'259877.071429,\n'
' "bw_dev" : 357.766676,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 64776,\n'
' "iops_max" : 65110,\n'
' "iops_mean" : '
'64969.214286,\n'
' "iops_stddev" : '
'89.484144,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 11.557143,\n'
' "sys_cpu" : 35.457143,\n'
' "ctx" : 454702,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 98.417728,\n'
' "50" : 1.522235,\n'
' "100" : 0.051679,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=84507: Tue '
'Oct 25 23:24:03 2022\n'
' read: IOPS=65.0k, BW=254MiB/s '
'(266MB/s)(1776MiB/7001msec)\n'
' slat (nsec): min=2314, '
'max=40516, avg=2487.81, '
'stdev=208.50\n'
' clat (nsec): min=351, '
'max=1341.1k, avg=12323.75, '
'stdev=2811.73\n'
' lat (usec): min=12, '
'max=1343, avg=14.81, stdev= 2.82\n'
' clat percentiles (nsec):\n'
' | 1.00th=[11840], '
'5.00th=[11840], 10.00th=[11840], '
'20.00th=[11840],\n'
' | 30.00th=[11968], '
'40.00th=[11968], 50.00th=[12096], '
'60.00th=[12096],\n'
' | 70.00th=[12096], '
'80.00th=[12224], 90.00th=[12224], '
'95.00th=[12480],\n'
' | 99.00th=[22144], '
'99.50th=[23680], 99.90th=[31616], '
'99.95th=[50432],\n'
' | 99.99th=[64256]\n'
' bw ( KiB/s): min=259104, '
'max=260440, per=100.00%, '
'avg=259877.07, stdev=357.77, '
'samples=14\n'
' iops : min=64776, '
'max=65110, avg=64969.21, '
'stdev=89.48, samples=14\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%\n'
' lat (usec) : 10=0.01%, '
'20=98.42%, 50=1.52%, 100=0.05%, '
'250=0.01%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=11.56%, '
'sys=35.46%, ctx=454702, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=454726,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=254MiB/s (266MB/s), '
'254MiB/s-254MiB/s '
'(266MB/s-266MB/s), io=1776MiB '
'(1863MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_12.output'),
'state': {'begin': 1666725954.0924244,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.803476095199585,
'end': 1666725954.8959005,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_12.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_12.state')},
'cmd_120': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733558,\n'
' "timestamp_ms" : '
'1666733558546,\n'
' "time" : "Tue Oct 25 23:32:38 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3217563648,\n'
' "io_kbytes" : 3142152,\n'
' "bw_bytes" : 459586294,\n'
' "bw" : 448814,\n'
' "iops" : 112203.542351,\n'
' "runtime" : 7001,\n'
' "total_ios" : 785537,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 661,\n'
' "max" : 7804,\n'
' "mean" : 916.427430,\n'
' "stddev" : 81.567851,\n'
' "N" : 785537\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 7284,\n'
' "max" : 134732,\n'
' "mean" : '
'16623.412960,\n'
' "stddev" : '
'3660.577461,\n'
' "N" : 785538,\n'
' "percentile" : {\n'
' "1.000000" : 13632,\n'
' "5.000000" : 13888,\n'
' "10.000000" : '
'14016,\n'
' "20.000000" : '
'14144,\n'
' "30.000000" : '
'14272,\n'
' "40.000000" : '
'14400,\n'
' "50.000000" : '
'17280,\n'
' "60.000000" : '
'17536,\n'
' "70.000000" : '
'17792,\n'
' "80.000000" : '
'18048,\n'
' "90.000000" : '
'18048,\n'
' "95.000000" : '
'19584,\n'
' "99.000000" : '
'30848,\n'
' "99.500000" : '
'34560,\n'
' "99.900000" : '
'51456,\n'
' "99.950000" : '
'65280,\n'
' "99.990000" : 79360\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11752,\n'
' "max" : 135514,\n'
' "mean" : '
'17539.840385,\n'
' "stddev" : '
'3659.715140,\n'
' "N" : 785538\n'
' },\n'
' "bw_min" : 447328,\n'
' "bw_max" : 451014,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'448941.000000,\n'
' "bw_dev" : 1118.741257,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 111832,\n'
' "iops_max" : 112753,\n'
' "iops_mean" : '
'112235.357143,\n'
' "iops_stddev" : '
'279.439991,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 2,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 99.983196,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 95.054466,\n'
' "50" : 4.836819,\n'
' "100" : 0.108334,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=86080: Tue Oct 25 23:32:38 '
'2022\n'
' read: IOPS=112k, BW=438MiB/s '
'(460MB/s)(3069MiB/7001msec)\n'
' slat (nsec): min=661, '
'max=7804, avg=916.43, '
'stdev=81.57\n'
' clat (usec): min=7, max=134, '
'avg=16.62, stdev= 3.66\n'
' lat (usec): min=11, '
'max=135, avg=17.54, stdev= 3.66\n'
' clat percentiles (nsec):\n'
' | 1.00th=[13632], '
'5.00th=[13888], 10.00th=[14016], '
'20.00th=[14144],\n'
' | 30.00th=[14272], '
'40.00th=[14400], '
'50.00th=[17280], '
'60.00th=[17536],\n'
' | 70.00th=[17792], '
'80.00th=[18048], '
'90.00th=[18048], '
'95.00th=[19584],\n'
' | 99.00th=[30848], '
'99.50th=[34560], '
'99.90th=[51456], '
'99.95th=[65280],\n'
' | 99.99th=[79360]\n'
' bw ( KiB/s): min=447328, '
'max=451014, per=100.00%, '
'avg=448941.00, stdev=1118.74, '
'samples=14\n'
' iops : min=111832, '
'max=112753, avg=112235.36, '
'stdev=279.44, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=95.05%, 50=4.84%, 100=0.11%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=2, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=785537,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=438MiB/s (460MB/s), '
'438MiB/s-438MiB/s '
'(460MB/s-460MB/s), io=3069MiB '
'(3218MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_120.output'),
'state': {'begin': 1666726469.1268544,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7891857624053955,
'end': 1666726469.9160402,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_120.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_120.state')},
'cmd_121': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_121.output'),
'state': {'begin': 1666726471.97713,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7857165336608887,
'end': 1666726472.7628465,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_121.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_121.state')},
'cmd_122': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_122.output'),
'state': {'begin': 1666726472.76527,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.503538608551025,
'end': 1666726484.2688086,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_122.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_122.state')},
'cmd_123': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733573,\n'
' "timestamp_ms" : '
'1666733573691,\n'
' "time" : "Tue Oct 25 23:32:53 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3401510912,\n'
' "io_kbytes" : 3321788,\n'
' "bw_bytes" : 485860721,\n'
' "bw" : 474473,\n'
' "iops" : 118618.197400,\n'
' "runtime" : 7001,\n'
' "total_ios" : 830446,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 671,\n'
' "max" : 11402,\n'
' "mean" : 967.528109,\n'
' "stddev" : '
'130.573892,\n'
' "N" : 830446\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 7574,\n'
' "max" : 176641,\n'
' "mean" : '
'15406.331199,\n'
' "stddev" : '
'3395.306415,\n'
' "N" : 830447,\n'
' "percentile" : {\n'
' "1.000000" : 13632,\n'
' "5.000000" : 13760,\n'
' "10.000000" : '
'13888,\n'
' "20.000000" : '
'14656,\n'
' "30.000000" : '
'14784,\n'
' "40.000000" : '
'14784,\n'
' "50.000000" : '
'14912,\n'
' "60.000000" : '
'15040,\n'
' "70.000000" : '
'15040,\n'
' "80.000000" : '
'15168,\n'
' "90.000000" : '
'15296,\n'
' "95.000000" : '
'15808,\n'
' "99.000000" : '
'31104,\n'
' "99.500000" : '
'36096,\n'
' "99.900000" : '
'51456,\n'
' "99.950000" : '
'65280,\n'
' "99.990000" : 82432\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10650,\n'
' "max" : 177563,\n'
' "mean" : '
'16373.859542,\n'
' "stddev" : '
'3397.897104,\n'
' "N" : 830447\n'
' },\n'
' "bw_min" : 473232,\n'
' "bw_max" : 475744,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'474607.785714,\n'
' "bw_dev" : 803.230801,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 118308,\n'
' "iops_max" : 118936,\n'
' "iops_mean" : '
'118651.928571,\n'
' "iops_stddev" : '
'200.807778,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 7,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 15.182565,\n'
' "2" : 84.817435,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 95.736147,\n'
' "50" : 4.161378,\n'
' "100" : 0.099224,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=86130: Tue Oct 25 23:32:53 '
'2022\n'
' read: IOPS=119k, BW=463MiB/s '
'(486MB/s)(3244MiB/7001msec)\n'
' slat (nsec): min=671, '
'max=11402, avg=967.53, '
'stdev=130.57\n'
' clat (usec): min=7, max=176, '
'avg=15.41, stdev= 3.40\n'
' lat (usec): min=10, '
'max=177, avg=16.37, stdev= 3.40\n'
' clat percentiles (nsec):\n'
' | 1.00th=[13632], '
'5.00th=[13760], 10.00th=[13888], '
'20.00th=[14656],\n'
' | 30.00th=[14784], '
'40.00th=[14784], '
'50.00th=[14912], '
'60.00th=[15040],\n'
' | 70.00th=[15040], '
'80.00th=[15168], '
'90.00th=[15296], '
'95.00th=[15808],\n'
' | 99.00th=[31104], '
'99.50th=[36096], '
'99.90th=[51456], '
'99.95th=[65280],\n'
' | 99.99th=[82432]\n'
' bw ( KiB/s): min=473232, '
'max=475744, per=100.00%, '
'avg=474607.79, stdev=803.23, '
'samples=14\n'
' iops : min=118308, '
'max=118936, avg=118651.93, '
'stdev=200.81, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=95.74%, 50=4.16%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=7, majf=0, '
'minf=0\n'
' IO depths : 1=15.2%, '
'2=84.8%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=830446,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=463MiB/s (486MB/s), '
'463MiB/s-463MiB/s '
'(486MB/s-486MB/s), io=3244MiB '
'(3402MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_123.output'),
'state': {'begin': 1666726484.271526,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7860622406005859,
'end': 1666726485.0575883,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_123.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_123.state')},
'cmd_124': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_124.output'),
'state': {'begin': 1666726487.1294892,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7789244651794434,
'end': 1666726487.9084136,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_124.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_124.state')},
'cmd_125': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_125.output'),
'state': {'begin': 1666726487.9108245,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.484877109527588,
'end': 1666726499.3957016,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_125.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_125.state')},
'cmd_126': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733588,\n'
' "timestamp_ms" : '
'1666733588818,\n'
' "time" : "Tue Oct 25 23:33:08 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3410411520,\n'
' "io_kbytes" : 3330480,\n'
' "bw_bytes" : 487132055,\n'
' "bw" : 475714,\n'
' "iops" : 118928.581631,\n'
' "runtime" : 7001,\n'
' "total_ios" : 832619,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 661,\n'
' "max" : 50916,\n'
' "mean" : 946.325844,\n'
' "stddev" : '
'169.673091,\n'
' "N" : 832619\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 881,\n'
' "max" : 1304386,\n'
' "mean" : '
'15275.264800,\n'
' "stddev" : '
'3935.692281,\n'
' "N" : 832620,\n'
' "percentile" : {\n'
' "1.000000" : 13504,\n'
' "5.000000" : 13632,\n'
' "10.000000" : '
'13760,\n'
' "20.000000" : '
'13888,\n'
' "30.000000" : '
'14656,\n'
' "40.000000" : '
'14784,\n'
' "50.000000" : '
'14784,\n'
' "60.000000" : '
'14912,\n'
' "70.000000" : '
'15040,\n'
' "80.000000" : '
'15040,\n'
' "90.000000" : '
'15296,\n'
' "95.000000" : '
'15808,\n'
' "99.000000" : '
'31104,\n'
' "99.500000" : '
'36608,\n'
' "99.900000" : '
'48896,\n'
' "99.950000" : '
'64768,\n'
' "99.990000" : 82432\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10650,\n'
' "max" : 1305568,\n'
' "mean" : '
'16221.590603,\n'
' "stddev" : '
'3939.658232,\n'
' "N" : 832620\n'
' },\n'
' "bw_min" : 474520,\n'
' "bw_max" : 477072,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'475836.000000,\n'
' "bw_dev" : 739.461759,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 118630,\n'
' "iops_max" : 119268,\n'
' "iops_mean" : '
'118959.000000,\n'
' "iops_stddev" : '
'184.865440,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 19,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 25.146916,\n'
' "2" : 74.853084,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 95.767812,\n'
' "50" : 4.132622,\n'
' "100" : 0.094521,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=86180: Tue Oct 25 23:33:08 '
'2022\n'
' read: IOPS=119k, BW=465MiB/s '
'(487MB/s)(3252MiB/7001msec)\n'
' slat (nsec): min=661, '
'max=50916, avg=946.33, '
'stdev=169.67\n'
' clat (nsec): min=881, '
'max=1304.4k, avg=15275.26, '
'stdev=3935.69\n'
' lat (usec): min=10, '
'max=1305, avg=16.22, stdev= '
'3.94\n'
' clat percentiles (nsec):\n'
' | 1.00th=[13504], '
'5.00th=[13632], 10.00th=[13760], '
'20.00th=[13888],\n'
' | 30.00th=[14656], '
'40.00th=[14784], '
'50.00th=[14784], '
'60.00th=[14912],\n'
' | 70.00th=[15040], '
'80.00th=[15040], '
'90.00th=[15296], '
'95.00th=[15808],\n'
' | 99.00th=[31104], '
'99.50th=[36608], '
'99.90th=[48896], '
'99.95th=[64768],\n'
' | 99.99th=[82432]\n'
' bw ( KiB/s): min=474520, '
'max=477072, per=100.00%, '
'avg=475836.00, stdev=739.46, '
'samples=14\n'
' iops : min=118630, '
'max=119268, avg=118959.00, '
'stdev=184.87, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.01%, 20=95.77%, '
'50=4.13%\n'
' lat (usec) : 100=0.09%, '
'250=0.01%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=19, majf=0, '
'minf=0\n'
' IO depths : 1=25.1%, '
'2=74.9%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=832619,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=465MiB/s (487MB/s), '
'465MiB/s-465MiB/s '
'(487MB/s-487MB/s), io=3252MiB '
'(3410MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_126.output'),
'state': {'begin': 1666726499.3984802,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8138327598571777,
'end': 1666726500.212313,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_126.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_126.state')},
'cmd_127': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_127.output'),
'state': {'begin': 1666726501.2081702,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7935519218444824,
'end': 1666726502.001722,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_127.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_127.state')},
'cmd_128': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_128.output'),
'state': {'begin': 1666726502.0037568,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring-reference_0.txt"',
'cwd': 'None',
'elapsed': 11.046199321746826,
'end': 1666726513.049956,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_128.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_128.state')},
'cmd_129': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=io_uring, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733602,\n'
' "timestamp_ms" : '
'1666733602544,\n'
' "time" : "Tue Oct 25 23:33:22 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2728185856,\n'
' "io_kbytes" : 2664244,\n'
' "bw_bytes" : 389685167,\n'
' "bw" : 380551,\n'
' "iops" : 95137.837452,\n'
' "runtime" : 7001,\n'
' "total_ios" : 666060,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2254,\n'
' "max" : 101570,\n'
' "mean" : 2532.068260,\n'
' "stddev" : '
'388.305482,\n'
' "N" : 666060\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 150,\n'
' "max" : 157766,\n'
' "mean" : '
'17905.455248,\n'
' "stddev" : '
'3291.383779,\n'
' "N" : 666061,\n'
' "percentile" : {\n'
' "1.000000" : 16768,\n'
' "5.000000" : 16768,\n'
' "10.000000" : '
'16768,\n'
' "20.000000" : '
'17024,\n'
' "30.000000" : '
'17024,\n'
' "40.000000" : '
'17024,\n'
' "50.000000" : '
'17280,\n'
' "60.000000" : '
'17536,\n'
' "70.000000" : '
'17536,\n'
' "80.000000" : '
'17536,\n'
' "90.000000" : '
'17792,\n'
' "95.000000" : '
'19840,\n'
' "99.000000" : '
'34048,\n'
' "99.500000" : '
'40192,\n'
' "99.900000" : '
'50432,\n'
' "99.950000" : '
'63232,\n'
' "99.990000" : 74240\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11642,\n'
' "max" : 160081,\n'
' "mean" : '
'20437.523182,\n'
' "stddev" : '
'3295.392638,\n'
' "N" : 666061\n'
' },\n'
' "bw_min" : 379696,\n'
' "bw_max" : 382128,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'380707.500000,\n'
' "bw_dev" : 705.250065,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 94924,\n'
' "iops_max" : 95532,\n'
' "iops_mean" : '
'95176.857143,\n'
' "iops_stddev" : '
'176.305180,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 12.700000,\n'
' "sys_cpu" : 41.028571,\n'
' "ctx" : 332854,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.016215,\n'
' "20" : 95.018317,\n'
' "50" : 4.859622,\n'
' "100" : 0.102093,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 965358,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 12002,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 12002,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=86219: Tue '
'Oct 25 23:33:22 2022\n'
' read: IOPS=95.1k, BW=372MiB/s '
'(390MB/s)(2602MiB/7001msec)\n'
' slat (nsec): min=2254, '
'max=101570, avg=2532.07, '
'stdev=388.31\n'
' clat (nsec): min=150, '
'max=157766, avg=17905.46, '
'stdev=3291.38\n'
' lat (usec): min=11, '
'max=160, avg=20.44, stdev= 3.30\n'
' clat percentiles (nsec):\n'
' | 1.00th=[16768], '
'5.00th=[16768], 10.00th=[16768], '
'20.00th=[17024],\n'
' | 30.00th=[17024], '
'40.00th=[17024], '
'50.00th=[17280], '
'60.00th=[17536],\n'
' | 70.00th=[17536], '
'80.00th=[17536], '
'90.00th=[17792], '
'95.00th=[19840],\n'
' | 99.00th=[34048], '
'99.50th=[40192], '
'99.90th=[50432], '
'99.95th=[63232],\n'
' | 99.99th=[74240]\n'
' bw ( KiB/s): min=379696, '
'max=382128, per=100.00%, '
'avg=380707.50, stdev=705.25, '
'samples=14\n'
' iops : min=94924, '
'max=95532, avg=95176.86, '
'stdev=176.31, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.02%, 20=95.02%, 50=4.86%, '
'100=0.10%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=12.70%, '
'sys=41.03%, ctx=332854, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=666060,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=372MiB/s (390MB/s), '
'372MiB/s-372MiB/s '
'(390MB/s-390MB/s), io=2602MiB '
'(2728MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=965358/0, '
'merge=0/0, ticks=12002/0, '
'in_queue=12002, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_129.output'),
'state': {'begin': 1666726513.051712,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7920506000518799,
'end': 1666726513.8437626,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_129.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_129.state')},
'cmd_13': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_13.output'),
'state': {'begin': 1666725955.8996396,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8654091358184814,
'end': 1666725956.7650487,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_13.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_13.state')},
'cmd_130': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_130.output'),
'state': {'begin': 1666726514.8286247,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7782762050628662,
'end': 1666726515.606901,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_130.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_130.state')},
'cmd_131': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_131.output'),
'state': {'begin': 1666726515.6089776,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring-reference_1.txt"',
'cwd': 'None',
'elapsed': 11.042916536331177,
'end': 1666726526.651894,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_131.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_131.state')},
'cmd_132': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=io_uring, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733616,\n'
' "timestamp_ms" : '
'1666733616144,\n'
' "time" : "Tue Oct 25 23:33:36 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2763063296,\n'
' "io_kbytes" : 2698304,\n'
' "bw_bytes" : 394666947,\n'
' "bw" : 385416,\n'
' "iops" : 96354.092273,\n'
' "runtime" : 7001,\n'
' "total_ios" : 674575,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2134,\n'
' "max" : 30237,\n'
' "mean" : 2411.264627,\n'
' "stddev" : '
'301.369070,\n'
' "N" : 674575\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 3416,\n'
' "max" : 152055,\n'
' "mean" : '
'17806.873363,\n'
' "stddev" : '
'3321.266952,\n'
' "N" : 674576,\n'
' "percentile" : {\n'
' "1.000000" : 16768,\n'
' "5.000000" : 16768,\n'
' "10.000000" : '
'16768,\n'
' "20.000000" : '
'16768,\n'
' "30.000000" : '
'17024,\n'
' "40.000000" : '
'17024,\n'
' "50.000000" : '
'17280,\n'
' "60.000000" : '
'17280,\n'
' "70.000000" : '
'17536,\n'
' "80.000000" : '
'17536,\n'
' "90.000000" : '
'17536,\n'
' "95.000000" : '
'20096,\n'
' "99.000000" : '
'34048,\n'
' "99.500000" : '
'40192,\n'
' "99.900000" : '
'53504,\n'
' "99.950000" : '
'65280,\n'
' "99.990000" : 76288\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11702,\n'
' "max" : 154269,\n'
' "mean" : '
'20218.137700,\n'
' "stddev" : '
'3319.032476,\n'
' "N" : 674576\n'
' },\n'
' "bw_min" : 384640,\n'
' "bw_max" : 386432,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'385564.928571,\n'
' "bw_dev" : 441.118241,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 96160,\n'
' "iops_max" : 96608,\n'
' "iops_mean" : '
'96391.142857,\n'
' "iops_stddev" : '
'110.190644,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 11.157143,\n'
' "sys_cpu" : 41.785714,\n'
' "ctx" : 337155,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 94.990624,\n'
' "50" : 4.873884,\n'
' "100" : 0.124226,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 966533,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 11927,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 11927,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=86257: Tue '
'Oct 25 23:33:36 2022\n'
' read: IOPS=96.4k, BW=376MiB/s '
'(395MB/s)(2635MiB/7001msec)\n'
' slat (nsec): min=2134, '
'max=30237, avg=2411.26, '
'stdev=301.37\n'
' clat (usec): min=3, max=152, '
'avg=17.81, stdev= 3.32\n'
' lat (usec): min=11, '
'max=154, avg=20.22, stdev= 3.32\n'
' clat percentiles (nsec):\n'
' | 1.00th=[16768], '
'5.00th=[16768], 10.00th=[16768], '
'20.00th=[16768],\n'
' | 30.00th=[17024], '
'40.00th=[17024], '
'50.00th=[17280], '
'60.00th=[17280],\n'
' | 70.00th=[17536], '
'80.00th=[17536], '
'90.00th=[17536], '
'95.00th=[20096],\n'
' | 99.00th=[34048], '
'99.50th=[40192], '
'99.90th=[53504], '
'99.95th=[65280],\n'
' | 99.99th=[76288]\n'
' bw ( KiB/s): min=384640, '
'max=386432, per=100.00%, '
'avg=385564.93, stdev=441.12, '
'samples=14\n'
' iops : min=96160, '
'max=96608, avg=96391.14, '
'stdev=110.19, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=94.99%, 50=4.87%, '
'100=0.12%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=11.16%, '
'sys=41.79%, ctx=337155, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=674575,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=376MiB/s (395MB/s), '
'376MiB/s-376MiB/s '
'(395MB/s-395MB/s), io=2635MiB '
'(2763MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=966533/0, '
'merge=0/0, ticks=11927/0, '
'in_queue=11927, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_132.output'),
'state': {'begin': 1666726526.658701,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7847790718078613,
'end': 1666726527.44348,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_132.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_132.state')},
'cmd_133': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_133.output'),
'state': {'begin': 1666726528.439404,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7807440757751465,
'end': 1666726529.220148,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_133.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_133.state')},
'cmd_134': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_134.output'),
'state': {'begin': 1666726529.222193,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring-reference_2.txt"',
'cwd': 'None',
'elapsed': 11.053741455078125,
'end': 1666726540.2759345,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_134.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_134.state')},
'cmd_135': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=io_uring, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733629,\n'
' "timestamp_ms" : '
'1666733629769,\n'
' "time" : "Tue Oct 25 23:33:49 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2769817600,\n'
' "io_kbytes" : 2704900,\n'
' "bw_bytes" : 395631709,\n'
' "bw" : 386359,\n'
' "iops" : 96589.630053,\n'
' "runtime" : 7001,\n'
' "total_ios" : 676224,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2174,\n'
' "max" : 56947,\n'
' "mean" : 2455.820520,\n'
' "stddev" : '
'313.047630,\n'
' "N" : 676224\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 3426,\n'
' "max" : 121939,\n'
' "mean" : '
'17701.656814,\n'
' "stddev" : '
'3232.617042,\n'
' "N" : 676225,\n'
' "percentile" : {\n'
' "1.000000" : 16512,\n'
' "5.000000" : 16768,\n'
' "10.000000" : '
'16768,\n'
' "20.000000" : '
'16768,\n'
' "30.000000" : '
'16768,\n'
' "40.000000" : '
'16768,\n'
' "50.000000" : '
'17280,\n'
' "60.000000" : '
'17280,\n'
' "70.000000" : '
'17280,\n'
' "80.000000" : '
'17280,\n'
' "90.000000" : '
'17536,\n'
' "95.000000" : '
'19840,\n'
' "99.000000" : '
'34048,\n'
' "99.500000" : '
'36608,\n'
' "99.900000" : '
'50432,\n'
' "99.950000" : '
'63744,\n'
' "99.990000" : 74240\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11822,\n'
' "max" : 124183,\n'
' "mean" : '
'20157.477096,\n'
' "stddev" : '
'3230.430762,\n'
' "N" : 676225\n'
' },\n'
' "bw_min" : 385176,\n'
' "bw_max" : 387294,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'386515.642857,\n'
' "bw_dev" : 702.422359,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 96294,\n'
' "iops_max" : 96823,\n'
' "iops_mean" : '
'96628.857143,\n'
' "iops_stddev" : '
'175.563207,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 10.471429,\n'
' "sys_cpu" : 42.771429,\n'
' "ctx" : 337987,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 94.997516,\n'
' "50" : 4.886694,\n'
' "100" : 0.107509,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 967584,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 11897,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 11897,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=86295: Tue '
'Oct 25 23:33:49 2022\n'
' read: IOPS=96.6k, BW=377MiB/s '
'(396MB/s)(2642MiB/7001msec)\n'
' slat (nsec): min=2174, '
'max=56947, avg=2455.82, '
'stdev=313.05\n'
' clat (usec): min=3, max=121, '
'avg=17.70, stdev= 3.23\n'
' lat (usec): min=11, '
'max=124, avg=20.16, stdev= 3.23\n'
' clat percentiles (nsec):\n'
' | 1.00th=[16512], '
'5.00th=[16768], 10.00th=[16768], '
'20.00th=[16768],\n'
' | 30.00th=[16768], '
'40.00th=[16768], '
'50.00th=[17280], '
'60.00th=[17280],\n'
' | 70.00th=[17280], '
'80.00th=[17280], '
'90.00th=[17536], '
'95.00th=[19840],\n'
' | 99.00th=[34048], '
'99.50th=[36608], '
'99.90th=[50432], '
'99.95th=[63744],\n'
' | 99.99th=[74240]\n'
' bw ( KiB/s): min=385176, '
'max=387294, per=100.00%, '
'avg=386515.64, stdev=702.42, '
'samples=14\n'
' iops : min=96294, '
'max=96823, avg=96628.86, '
'stdev=175.56, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=95.00%, 50=4.89%, '
'100=0.11%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=10.47%, '
'sys=42.77%, ctx=337987, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=676224,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=377MiB/s (396MB/s), '
'377MiB/s-377MiB/s '
'(396MB/s-396MB/s), io=2642MiB '
'(2770MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=967584/0, '
'merge=0/0, ticks=11897/0, '
'in_queue=11897, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_135.output'),
'state': {'begin': 1666726540.279404,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7870666980743408,
'end': 1666726541.0664706,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_135.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_135.state')},
'cmd_136': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_136.output'),
'state': {'begin': 1666726543.085222,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.7838060855865479,
'end': 1666726543.869028,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_136.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_136.state')},
'cmd_137': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_137.output'),
'state': {'begin': 1666726543.8709989,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt"',
'cwd': 'None',
'elapsed': 11.33250117301941,
'end': 1666726555.2035,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_137.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_137.state')},
'cmd_138': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733644,\n'
' "timestamp_ms" : '
'1666733644623,\n'
' "time" : "Tue Oct 25 23:34:04 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3403218944,\n'
' "io_kbytes" : 3323456,\n'
' "bw_bytes" : 486104691,\n'
' "bw" : 474711,\n'
' "iops" : 118677.760320,\n'
' "runtime" : 7001,\n'
' "total_ios" : 830863,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 13305,\n'
' "mean" : 152.035344,\n'
' "stddev" : 44.246764,\n'
' "N" : 830863\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 11011,\n'
' "max" : 125345,\n'
' "mean" : '
'16229.591323,\n'
' "stddev" : '
'3439.794272,\n'
' "N" : 830863,\n'
' "percentile" : {\n'
' "1.000000" : 15168,\n'
' "5.000000" : 15168,\n'
' "10.000000" : '
'15296,\n'
' "20.000000" : '
'15296,\n'
' "30.000000" : '
'15424,\n'
' "40.000000" : '
'15552,\n'
' "50.000000" : '
'15552,\n'
' "60.000000" : '
'15680,\n'
' "70.000000" : '
'15680,\n'
' "80.000000" : '
'15808,\n'
' "90.000000" : '
'15936,\n'
' "95.000000" : '
'16512,\n'
' "99.000000" : '
'31872,\n'
' "99.500000" : '
'37632,\n'
' "99.900000" : '
'49920,\n'
' "99.950000" : '
'66048,\n'
' "99.990000" : 84480\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11191,\n'
' "max" : 125525,\n'
' "mean" : '
'16381.626666,\n'
' "stddev" : '
'3441.514411,\n'
' "N" : 830863\n'
' },\n'
' "bw_min" : 473656,\n'
' "bw_max" : 476112,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'474806.214286,\n'
' "bw_dev" : 872.970718,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 118414,\n'
' "iops_max" : 119028,\n'
' "iops_mean" : '
'118701.214286,\n'
' "iops_stddev" : '
'218.461818,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 77.185714,\n'
' "sys_cpu" : 22.785714,\n'
' "ctx" : 16,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 49.969128,\n'
' "2" : 50.030872,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 95.630567,\n'
' "50" : 4.272425,\n'
' "100" : 0.096526,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=86344: Tue '
'Oct 25 23:34:04 2022\n'
' read: IOPS=119k, BW=464MiB/s '
'(486MB/s)(3246MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=13305, avg=152.04, '
'stdev=44.25\n'
' clat (usec): min=11, '
'max=125, avg=16.23, stdev= 3.44\n'
' lat (usec): min=11, '
'max=125, avg=16.38, stdev= 3.44\n'
' clat percentiles (nsec):\n'
' | 1.00th=[15168], '
'5.00th=[15168], 10.00th=[15296], '
'20.00th=[15296],\n'
' | 30.00th=[15424], '
'40.00th=[15552], '
'50.00th=[15552], '
'60.00th=[15680],\n'
' | 70.00th=[15680], '
'80.00th=[15808], '
'90.00th=[15936], '
'95.00th=[16512],\n'
' | 99.00th=[31872], '
'99.50th=[37632], '
'99.90th=[49920], '
'99.95th=[66048],\n'
' | 99.99th=[84480]\n'
' bw ( KiB/s): min=473656, '
'max=476112, per=100.00%, '
'avg=474806.21, stdev=872.97, '
'samples=14\n'
' iops : min=118414, '
'max=119028, avg=118701.21, '
'stdev=218.46, samples=14\n'
' lat (usec) : 20=95.63%, '
'50=4.27%, 100=0.10%, 250=0.01%\n'
' cpu : usr=77.19%, '
'sys=22.79%, ctx=16, majf=0, '
'minf=0\n'
' IO depths : 1=50.0%, '
'2=50.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=830863,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=464MiB/s (486MB/s), '
'464MiB/s-464MiB/s '
'(486MB/s-486MB/s), io=3246MiB '
'(3403MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_138.output'),
'state': {'begin': 1666726555.2066221,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.7880933284759521,
'end': 1666726555.9947155,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_138.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_138.state')},
'cmd_139': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_139.output'),
'state': {'begin': 1666726558.1017845,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7905898094177246,
'end': 1666726558.8923743,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_139.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_139.state')},
'cmd_14': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_14.output'),
'state': {'begin': 1666725956.7671995,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.056231498718262,
'end': 1666725967.823431,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_14.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_14.state')},
'cmd_140': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_140.output'),
'state': {'begin': 1666726558.8945546,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt"',
'cwd': 'None',
'elapsed': 11.452940225601196,
'end': 1666726570.3474948,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_140.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_140.state')},
'cmd_141': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733659,\n'
' "timestamp_ms" : '
'1666733659767,\n'
' "time" : "Tue Oct 25 23:34:19 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3699757056,\n'
' "io_kbytes" : 3613044,\n'
' "bw_bytes" : 528461227,\n'
' "bw" : 516075,\n'
' "iops" : 129018.711613,\n'
' "runtime" : 7001,\n'
' "total_ios" : 903260,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 13215,\n'
' "mean" : 159.408986,\n'
' "stddev" : 56.703624,\n'
' "N" : 903260\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 9217,\n'
' "max" : 1316439,\n'
' "mean" : '
'14838.287266,\n'
' "stddev" : '
'4093.044611,\n'
' "N" : 903260,\n'
' "percentile" : {\n'
' "1.000000" : 13888,\n'
' "5.000000" : 13888,\n'
' "10.000000" : '
'13888,\n'
' "20.000000" : '
'14016,\n'
' "30.000000" : '
'14016,\n'
' "40.000000" : '
'14144,\n'
' "50.000000" : '
'14144,\n'
' "60.000000" : '
'14272,\n'
' "70.000000" : '
'14272,\n'
' "80.000000" : '
'14400,\n'
' "90.000000" : '
'14400,\n'
' "95.000000" : '
'15168,\n'
' "99.000000" : '
'31616,\n'
' "99.500000" : '
'39168,\n'
' "99.900000" : '
'48384,\n'
' "99.950000" : '
'66048,\n'
' "99.990000" : 84480\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9407,\n'
' "max" : 1316629,\n'
' "mean" : '
'14997.696252,\n'
' "stddev" : '
'4094.615509,\n'
' "N" : 903260\n'
' },\n'
' "bw_min" : 514352,\n'
' "bw_max" : 517672,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'516195.357143,\n'
' "bw_dev" : 856.680679,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 128588,\n'
' "iops_max" : 129418,\n'
' "iops_mean" : '
'129048.785714,\n'
' "iops_stddev" : '
'214.201264,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 68.600000,\n'
' "sys_cpu" : 31.357143,\n'
' "ctx" : 50,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 49.986936,\n'
' "2" : 50.013064,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.020924,\n'
' "20" : 96.007240,\n'
' "50" : 3.875518,\n'
' "100" : 0.095654,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=86393: Tue '
'Oct 25 23:34:19 2022\n'
' read: IOPS=129k, BW=504MiB/s '
'(528MB/s)(3528MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=13215, avg=159.41, '
'stdev=56.70\n'
' clat (usec): min=9, '
'max=1316, avg=14.84, stdev= '
'4.09\n'
' lat (usec): min=9, '
'max=1316, avg=15.00, stdev= '
'4.09\n'
' clat percentiles (nsec):\n'
' | 1.00th=[13888], '
'5.00th=[13888], 10.00th=[13888], '
'20.00th=[14016],\n'
' | 30.00th=[14016], '
'40.00th=[14144], '
'50.00th=[14144], '
'60.00th=[14272],\n'
' | 70.00th=[14272], '
'80.00th=[14400], '
'90.00th=[14400], '
'95.00th=[15168],\n'
' | 99.00th=[31616], '
'99.50th=[39168], '
'99.90th=[48384], '
'99.95th=[66048],\n'
' | 99.99th=[84480]\n'
' bw ( KiB/s): min=514352, '
'max=517672, per=100.00%, '
'avg=516195.36, stdev=856.68, '
'samples=14\n'
' iops : min=128588, '
'max=129418, avg=129048.79, '
'stdev=214.20, samples=14\n'
' lat (usec) : 10=0.02%, '
'20=96.01%, 50=3.88%, 100=0.10%, '
'250=0.01%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=68.60%, '
'sys=31.36%, ctx=50, majf=0, '
'minf=0\n'
' IO depths : 1=50.0%, '
'2=50.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=903260,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=504MiB/s (528MB/s), '
'504MiB/s-504MiB/s '
'(528MB/s-528MB/s), io=3528MiB '
'(3700MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_141.output'),
'state': {'begin': 1666726570.3504097,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7888486385345459,
'end': 1666726571.1392584,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_141.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_141.state')},
'cmd_142': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_142.output'),
'state': {'begin': 1666726573.2099755,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.794795036315918,
'end': 1666726574.0047705,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_142.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_142.state')},
'cmd_143': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_143.output'),
'state': {'begin': 1666726574.0118794,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt"',
'cwd': 'None',
'elapsed': 11.472287893295288,
'end': 1666726585.4841673,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_143.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_143.state')},
'cmd_144': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733674,\n'
' "timestamp_ms" : '
'1666733674905,\n'
' "time" : "Tue Oct 25 23:34:34 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3672137728,\n'
' "io_kbytes" : 3586072,\n'
' "bw_bytes" : 524516173,\n'
' "bw" : 512222,\n'
' "iops" : 128055.563491,\n'
' "runtime" : 7001,\n'
' "total_ios" : 896517,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 100,\n'
' "max" : 16821,\n'
' "mean" : 151.655918,\n'
' "stddev" : 54.676795,\n'
' "N" : 896517\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 9408,\n'
' "max" : 102191,\n'
' "mean" : '
'14971.500404,\n'
' "stddev" : '
'3599.731107,\n'
' "N" : 896517,\n'
' "percentile" : {\n'
' "1.000000" : 14016,\n'
' "5.000000" : 14016,\n'
' "10.000000" : '
'14144,\n'
' "20.000000" : '
'14144,\n'
' "30.000000" : '
'14144,\n'
' "40.000000" : '
'14272,\n'
' "50.000000" : '
'14272,\n'
' "60.000000" : '
'14400,\n'
' "70.000000" : '
'14400,\n'
' "80.000000" : '
'14400,\n'
' "90.000000" : '
'14528,\n'
' "95.000000" : '
'15296,\n'
' "99.000000" : '
'31616,\n'
' "99.500000" : '
'39168,\n'
' "99.900000" : '
'53504,\n'
' "99.950000" : '
'67072,\n'
' "99.990000" : 82432\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9578,\n'
' "max" : 102372,\n'
' "mean" : '
'15123.156323,\n'
' "stddev" : '
'3601.418454,\n'
' "N" : 896517\n'
' },\n'
' "bw_min" : 511288,\n'
' "bw_max" : 513200,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'512352.071429,\n'
' "bw_dev" : 676.281287,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 127820,\n'
' "iops_max" : 128300,\n'
' "iops_mean" : '
'128088.000000,\n'
' "iops_stddev" : '
'169.357156,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 64.900000,\n'
' "sys_cpu" : 35.071429,\n'
' "ctx" : 16,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 49.996933,\n'
' "2" : 50.003067,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 95.756801,\n'
' "50" : 4.139018,\n'
' "100" : 0.101950,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=86443: Tue '
'Oct 25 23:34:34 2022\n'
' read: IOPS=128k, BW=500MiB/s '
'(525MB/s)(3502MiB/7001msec)\n'
' slat (nsec): min=100, '
'max=16821, avg=151.66, '
'stdev=54.68\n'
' clat (usec): min=9, max=102, '
'avg=14.97, stdev= 3.60\n'
' lat (usec): min=9, max=102, '
'avg=15.12, stdev= 3.60\n'
' clat percentiles (nsec):\n'
' | 1.00th=[14016], '
'5.00th=[14016], 10.00th=[14144], '
'20.00th=[14144],\n'
' | 30.00th=[14144], '
'40.00th=[14272], '
'50.00th=[14272], '
'60.00th=[14400],\n'
' | 70.00th=[14400], '
'80.00th=[14400], '
'90.00th=[14528], '
'95.00th=[15296],\n'
' | 99.00th=[31616], '
'99.50th=[39168], '
'99.90th=[53504], '
'99.95th=[67072],\n'
' | 99.99th=[82432]\n'
' bw ( KiB/s): min=511288, '
'max=513200, per=100.00%, '
'avg=512352.07, stdev=676.28, '
'samples=14\n'
' iops : min=127820, '
'max=128300, avg=128088.00, '
'stdev=169.36, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=95.76%, 50=4.14%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=64.90%, '
'sys=35.07%, ctx=16, majf=0, '
'minf=0\n'
' IO depths : 1=50.0%, '
'2=50.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=896517,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=500MiB/s (525MB/s), '
'500MiB/s-500MiB/s '
'(525MB/s-525MB/s), io=3502MiB '
'(3672MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_144.output'),
'state': {'begin': 1666726585.4879289,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.7787735462188721,
'end': 1666726586.2667024,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_144.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_144.state')},
'cmd_145': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_145.output'),
'state': {'begin': 1666726588.3478794,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7863411903381348,
'end': 1666726589.1342206,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_145.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_145.state')},
'cmd_146': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_146.output'),
'state': {'begin': 1666726589.1363273,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.435516595840454,
'end': 1666726600.5718439,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_146.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_146.state')},
'cmd_147': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733689,\n'
' "timestamp_ms" : '
'1666733689991,\n'
' "time" : "Tue Oct 25 23:34:49 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3760844800,\n'
' "io_kbytes" : 3672700,\n'
' "bw_bytes" : 537186801,\n'
' "bw" : 524596,\n'
' "iops" : 131148.978717,\n'
' "runtime" : 7001,\n'
' "total_ios" : 918174,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1643,\n'
' "max" : 15720,\n'
' "mean" : 1814.378433,\n'
' "stddev" : '
'122.167456,\n'
' "N" : 918174\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 261,\n'
' "max" : 95289,\n'
' "mean" : '
'13203.321858,\n'
' "stddev" : '
'4291.070630,\n'
' "N" : 918175,\n'
' "percentile" : {\n'
' "1.000000" : 10048,\n'
' "5.000000" : 10176,\n'
' "10.000000" : '
'10176,\n'
' "20.000000" : '
'10176,\n'
' "30.000000" : '
'10304,\n'
' "40.000000" : '
'10304,\n'
' "50.000000" : '
'14528,\n'
' "60.000000" : '
'14656,\n'
' "70.000000" : '
'14656,\n'
' "80.000000" : '
'14784,\n'
' "90.000000" : '
'14912,\n'
' "95.000000" : '
'15552,\n'
' "99.000000" : '
'31616,\n'
' "99.500000" : '
'37120,\n'
' "99.900000" : '
'44800,\n'
' "99.950000" : '
'62720,\n'
' "99.990000" : 76288\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9618,\n'
' "max" : 97092,\n'
' "mean" : '
'15017.700203,\n'
' "stddev" : '
'4292.850203,\n'
' "N" : 918175\n'
' },\n'
' "bw_min" : 523296,\n'
' "bw_max" : 526613,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'524883.928571,\n'
' "bw_dev" : 955.591357,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 130824,\n'
' "iops_max" : 131653,\n'
' "iops_mean" : '
'131220.928571,\n'
' "iops_stddev" : '
'238.846737,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 11.700000,\n'
' "sys_cpu" : 88.271429,\n'
' "ctx" : 13,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.690174,\n'
' "20" : 95.137632,\n'
' "50" : 4.081253,\n'
' "100" : 0.088110,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=86494: Tue '
'Oct 25 23:34:49 2022\n'
' read: IOPS=131k, BW=512MiB/s '
'(537MB/s)(3587MiB/7001msec)\n'
' slat (nsec): min=1643, '
'max=15720, avg=1814.38, '
'stdev=122.17\n'
' clat (nsec): min=261, '
'max=95289, avg=13203.32, '
'stdev=4291.07\n'
' lat (nsec): min=9618, '
'max=97092, avg=15017.70, '
'stdev=4292.85\n'
' clat percentiles (nsec):\n'
' | 1.00th=[10048], '
'5.00th=[10176], 10.00th=[10176], '
'20.00th=[10176],\n'
' | 30.00th=[10304], '
'40.00th=[10304], '
'50.00th=[14528], '
'60.00th=[14656],\n'
' | 70.00th=[14656], '
'80.00th=[14784], '
'90.00th=[14912], '
'95.00th=[15552],\n'
' | 99.00th=[31616], '
'99.50th=[37120], '
'99.90th=[44800], '
'99.95th=[62720],\n'
' | 99.99th=[76288]\n'
' bw ( KiB/s): min=523296, '
'max=526613, per=100.00%, '
'avg=524883.93, stdev=955.59, '
'samples=14\n'
' iops : min=130824, '
'max=131653, avg=131220.93, '
'stdev=238.85, samples=14\n'
' lat (nsec) : 500=0.01%, '
'1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.69%, 20=95.14%, '
'50=4.08%\n'
' lat (usec) : 100=0.09%\n'
' cpu : usr=11.70%, '
'sys=88.27%, ctx=13, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=918174,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=512MiB/s (537MB/s), '
'512MiB/s-512MiB/s '
'(537MB/s-537MB/s), io=3587MiB '
'(3761MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_147.output'),
'state': {'begin': 1666726600.5745914,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7963974475860596,
'end': 1666726601.3709888,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_147.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_147.state')},
'cmd_148': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_148.output'),
'state': {'begin': 1666726603.3925414,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7927045822143555,
'end': 1666726604.185246,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_148.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_148.state')},
'cmd_149': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_149.output'),
'state': {'begin': 1666726604.1870391,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.46225118637085,
'end': 1666726615.6492903,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_149.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_149.state')},
'cmd_15': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733057,\n'
' "timestamp_ms" : '
'1666733057326,\n'
' "time" : "Tue Oct 25 23:24:17 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1745297408,\n'
' "io_kbytes" : 1704392,\n'
' "bw_bytes" : 249292587,\n'
' "bw" : 243449,\n'
' "iops" : 60862.448222,\n'
' "runtime" : 7001,\n'
' "total_ios" : 426098,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2294,\n'
' "max" : 44142,\n'
' "mean" : 2645.758187,\n'
' "stddev" : 205.940659,\n'
' "N" : 426098\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 351,\n'
' "max" : 105057,\n'
' "mean" : 13250.876676,\n'
' "stddev" : '
'2201.080057,\n'
' "N" : 426098,\n'
' "percentile" : {\n'
' "1.000000" : 12096,\n'
' "5.000000" : 12096,\n'
' "10.000000" : 12096,\n'
' "20.000000" : 12224,\n'
' "30.000000" : 12352,\n'
' "40.000000" : 13248,\n'
' "50.000000" : 13376,\n'
' "60.000000" : 13376,\n'
' "70.000000" : 13504,\n'
' "80.000000" : 13504,\n'
' "90.000000" : 13504,\n'
' "95.000000" : 13632,\n'
' "99.000000" : 22912,\n'
' "99.500000" : 29312,\n'
' "99.900000" : 38656,\n'
' "99.950000" : 52992,\n'
' "99.990000" : 69120\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 14187,\n'
' "max" : 114945,\n'
' "mean" : 15896.634863,\n'
' "stddev" : '
'2223.085816,\n'
' "N" : 426098\n'
' },\n'
' "bw_min" : 235512,\n'
' "bw_max" : 257448,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'243501.285714,\n'
' "bw_dev" : 9024.250392,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 58878,\n'
' "iops_max" : 64362,\n'
' "iops_mean" : '
'60875.285714,\n'
' "iops_stddev" : '
'2256.088870,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 9.300000,\n'
' "sys_cpu" : 34.514286,\n'
' "ctx" : 426091,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 98.344982,\n'
' "50" : 1.589306,\n'
' "100" : 0.061488,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=84545: Tue '
'Oct 25 23:24:17 2022\n'
' read: IOPS=60.9k, BW=238MiB/s '
'(249MB/s)(1664MiB/7001msec)\n'
' slat (nsec): min=2294, '
'max=44142, avg=2645.76, '
'stdev=205.94\n'
' clat (nsec): min=351, '
'max=105057, avg=13250.88, '
'stdev=2201.08\n'
' lat (usec): min=14, max=114, '
'avg=15.90, stdev= 2.22\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12096], '
'5.00th=[12096], 10.00th=[12096], '
'20.00th=[12224],\n'
' | 30.00th=[12352], '
'40.00th=[13248], 50.00th=[13376], '
'60.00th=[13376],\n'
' | 70.00th=[13504], '
'80.00th=[13504], 90.00th=[13504], '
'95.00th=[13632],\n'
' | 99.00th=[22912], '
'99.50th=[29312], 99.90th=[38656], '
'99.95th=[52992],\n'
' | 99.99th=[69120]\n'
' bw ( KiB/s): min=235512, '
'max=257448, per=100.00%, '
'avg=243501.29, stdev=9024.25, '
'samples=14\n'
' iops : min=58878, '
'max=64362, avg=60875.29, '
'stdev=2256.09, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 10=0.01%, '
'20=98.34%, 50=1.59%, 100=0.06%, '
'250=0.01%\n'
' cpu : usr=9.30%, '
'sys=34.51%, ctx=426091, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=426098,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=238MiB/s (249MB/s), '
'238MiB/s-238MiB/s '
'(249MB/s-249MB/s), io=1664MiB '
'(1745MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_15.output'),
'state': {'begin': 1666725967.826345,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7927157878875732,
'end': 1666725968.6190608,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_15.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_15.state')},
'cmd_150': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733705,\n'
' "timestamp_ms" : '
'1666733705068,\n'
' "time" : "Tue Oct 25 23:35:05 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3756642304,\n'
' "io_kbytes" : 3668596,\n'
' "bw_bytes" : 536586531,\n'
' "bw" : 524010,\n'
' "iops" : 131002.428225,\n'
' "runtime" : 7001,\n'
' "total_ios" : 917148,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1643,\n'
' "max" : 19396,\n'
' "mean" : 1811.534450,\n'
' "stddev" : '
'120.874810,\n'
' "N" : 917148\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 591,\n'
' "max" : 102302,\n'
' "mean" : '
'13208.573923,\n'
' "stddev" : '
'4374.320123,\n'
' "N" : 917149,\n'
' "percentile" : {\n'
' "1.000000" : 10048,\n'
' "5.000000" : 10176,\n'
' "10.000000" : '
'10176,\n'
' "20.000000" : '
'10176,\n'
' "30.000000" : '
'10304,\n'
' "40.000000" : '
'10304,\n'
' "50.000000" : '
'14528,\n'
' "60.000000" : '
'14656,\n'
' "70.000000" : '
'14656,\n'
' "80.000000" : '
'14784,\n'
' "90.000000" : '
'14912,\n'
' "95.000000" : '
'15552,\n'
' "99.000000" : '
'31872,\n'
' "99.500000" : '
'37120,\n'
' "99.900000" : '
'50432,\n'
' "99.950000" : '
'65280,\n'
' "99.990000" : 77312\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9618,\n'
' "max" : 104076,\n'
' "mean" : '
'15020.108310,\n'
' "stddev" : '
'4377.112921,\n'
' "N" : 917149\n'
' },\n'
' "bw_min" : 522248,\n'
' "bw_max" : 526256,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'524297.500000,\n'
' "bw_dev" : 1261.415673,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 130562,\n'
' "iops_max" : 131564,\n'
' "iops_mean" : '
'131074.214286,\n'
' "iops_stddev" : '
'315.322101,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 10.914286,\n'
' "sys_cpu" : 89.071429,\n'
' "ctx" : 17,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.643953,\n'
' "20" : 95.156507,\n'
' "50" : 4.090834,\n'
' "100" : 0.104999,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=86544: Tue '
'Oct 25 23:35:05 2022\n'
' read: IOPS=131k, BW=512MiB/s '
'(537MB/s)(3583MiB/7001msec)\n'
' slat (nsec): min=1643, '
'max=19396, avg=1811.53, '
'stdev=120.87\n'
' clat (nsec): min=591, '
'max=102302, avg=13208.57, '
'stdev=4374.32\n'
' lat (usec): min=9, max=104, '
'avg=15.02, stdev= 4.38\n'
' clat percentiles (nsec):\n'
' | 1.00th=[10048], '
'5.00th=[10176], 10.00th=[10176], '
'20.00th=[10176],\n'
' | 30.00th=[10304], '
'40.00th=[10304], '
'50.00th=[14528], '
'60.00th=[14656],\n'
' | 70.00th=[14656], '
'80.00th=[14784], '
'90.00th=[14912], '
'95.00th=[15552],\n'
' | 99.00th=[31872], '
'99.50th=[37120], '
'99.90th=[50432], '
'99.95th=[65280],\n'
' | 99.99th=[77312]\n'
' bw ( KiB/s): min=522248, '
'max=526256, per=100.00%, '
'avg=524297.50, stdev=1261.42, '
'samples=14\n'
' iops : min=130562, '
'max=131564, avg=131074.21, '
'stdev=315.32, samples=14\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.64%, 20=95.16%, '
'50=4.09%\n'
' lat (usec) : 100=0.10%, '
'250=0.01%\n'
' cpu : usr=10.91%, '
'sys=89.07%, ctx=17, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=917148,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=512MiB/s (537MB/s), '
'512MiB/s-512MiB/s '
'(537MB/s-537MB/s), io=3583MiB '
'(3757MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_150.output'),
'state': {'begin': 1666726615.6520617,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7919247150421143,
'end': 1666726616.4439864,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_150.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_150.state')},
'cmd_151': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_151.output'),
'state': {'begin': 1666726618.4636512,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7874436378479004,
'end': 1666726619.2510948,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_151.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_151.state')},
'cmd_152': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_152.output'),
'state': {'begin': 1666726619.2533228,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.456940650939941,
'end': 1666726630.7102635,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_152.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_152.state')},
'cmd_153': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733720,\n'
' "timestamp_ms" : '
'1666733720129,\n'
' "time" : "Tue Oct 25 23:35:20 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3880730624,\n'
' "io_kbytes" : 3789776,\n'
' "bw_bytes" : 554310901,\n'
' "bw" : 541319,\n'
' "iops" : 135329.667190,\n'
' "runtime" : 7001,\n'
' "total_ios" : 947443,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1492,\n'
' "max" : 15409,\n'
' "mean" : 1579.608812,\n'
' "stddev" : '
'127.933694,\n'
' "N" : 947443\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 270,\n'
' "max" : 118652,\n'
' "mean" : '
'12956.637074,\n'
' "stddev" : '
'3894.360504,\n'
' "N" : 947444,\n'
' "percentile" : {\n'
' "1.000000" : 10048,\n'
' "5.000000" : 10176,\n'
' "10.000000" : '
'10176,\n'
' "20.000000" : '
'11968,\n'
' "30.000000" : '
'12096,\n'
' "40.000000" : '
'12224,\n'
' "50.000000" : '
'12224,\n'
' "60.000000" : '
'12352,\n'
' "70.000000" : '
'12352,\n'
' "80.000000" : '
'14144,\n'
' "90.000000" : '
'14400,\n'
' "95.000000" : '
'14656,\n'
' "99.000000" : '
'30592,\n'
' "99.500000" : '
'37120,\n'
' "99.900000" : '
'52480,\n'
' "99.950000" : '
'64768,\n'
' "99.990000" : 75264\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9178,\n'
' "max" : 120225,\n'
' "mean" : '
'14536.245858,\n'
' "stddev" : '
'3892.786982,\n'
' "N" : 947444\n'
' },\n'
' "bw_min" : 540352,\n'
' "bw_max" : 542264,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'541450.857143,\n'
' "bw_dev" : 675.183719,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 135088,\n'
' "iops_max" : 135566,\n'
' "iops_mean" : '
'135362.571429,\n'
' "iops_stddev" : '
'168.735439,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 9.314286,\n'
' "sys_cpu" : 90.671429,\n'
' "ctx" : 7,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.089082,\n'
' "20" : 96.324423,\n'
' "50" : 3.473665,\n'
' "100" : 0.109769,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=86593: Tue '
'Oct 25 23:35:20 2022\n'
' read: IOPS=135k, BW=529MiB/s '
'(554MB/s)(3701MiB/7001msec)\n'
' slat (nsec): min=1492, '
'max=15409, avg=1579.61, '
'stdev=127.93\n'
' clat (nsec): min=270, '
'max=118652, avg=12956.64, '
'stdev=3894.36\n'
' lat (usec): min=9, max=120, '
'avg=14.54, stdev= 3.89\n'
' clat percentiles (nsec):\n'
' | 1.00th=[10048], '
'5.00th=[10176], 10.00th=[10176], '
'20.00th=[11968],\n'
' | 30.00th=[12096], '
'40.00th=[12224], '
'50.00th=[12224], '
'60.00th=[12352],\n'
' | 70.00th=[12352], '
'80.00th=[14144], '
'90.00th=[14400], '
'95.00th=[14656],\n'
' | 99.00th=[30592], '
'99.50th=[37120], '
'99.90th=[52480], '
'99.95th=[64768],\n'
' | 99.99th=[75264]\n'
' bw ( KiB/s): min=540352, '
'max=542264, per=100.00%, '
'avg=541450.86, stdev=675.18, '
'samples=14\n'
' iops : min=135088, '
'max=135566, avg=135362.57, '
'stdev=168.74, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.09%, 20=96.32%, 50=3.47%, '
'100=0.11%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=9.31%, '
'sys=90.67%, ctx=7, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=947443,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=529MiB/s (554MB/s), '
'529MiB/s-529MiB/s '
'(554MB/s-554MB/s), io=3701MiB '
'(3881MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_153.output'),
'state': {'begin': 1666726630.7130911,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7981386184692383,
'end': 1666726631.5112298,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_153.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_153.state')},
'cmd_154': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_154.output'),
'state': {'begin': 1666726632.5046132,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7883801460266113,
'end': 1666726633.2929933,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_154.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_154.state')},
'cmd_155': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_155.output'),
'state': {'begin': 1666726633.294968,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring-xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.063021183013916,
'end': 1666726644.357989,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_155.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_155.state')},
'cmd_156': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733733,\n'
' "timestamp_ms" : '
'1666733733849,\n'
' "time" : "Tue Oct 25 23:35:33 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2654441472,\n'
' "io_kbytes" : 2592228,\n'
' "bw_bytes" : 379151760,\n'
' "bw" : 370265,\n'
' "iops" : 92566.061991,\n'
' "runtime" : 7001,\n'
' "total_ios" : 648055,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2304,\n'
' "max" : 769522,\n'
' "mean" : 2713.917901,\n'
' "stddev" : '
'997.462060,\n'
' "N" : 648056\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 341,\n'
' "max" : 775153,\n'
' "mean" : '
'18398.808790,\n'
' "stddev" : '
'3486.347443,\n'
' "N" : 648056,\n'
' "percentile" : {\n'
' "1.000000" : 16768,\n'
' "5.000000" : 16768,\n'
' "10.000000" : '
'17024,\n'
' "20.000000" : '
'17280,\n'
' "30.000000" : '
'17536,\n'
' "40.000000" : '
'17792,\n'
' "50.000000" : '
'17792,\n'
' "60.000000" : '
'18048,\n'
' "70.000000" : '
'18048,\n'
' "80.000000" : '
'18304,\n'
' "90.000000" : '
'18304,\n'
' "95.000000" : '
'20864,\n'
' "99.000000" : '
'34560,\n'
' "99.500000" : '
'43776,\n'
' "99.900000" : '
'50432,\n'
' "99.950000" : '
'63232,\n'
' "99.990000" : 75264\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12253,\n'
' "max" : 792535,\n'
' "mean" : '
'21112.726690,\n'
' "stddev" : '
'3619.369648,\n'
' "N" : 648056\n'
' },\n'
' "bw_min" : 365080,\n'
' "bw_max" : 384096,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'370353.714286,\n'
' "bw_dev" : 8054.404657,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 91270,\n'
' "iops_max" : 96024,\n'
' "iops_mean" : '
'92588.428571,\n'
' "iops_stddev" : '
'2013.601164,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 11.171429,\n'
' "sys_cpu" : 40.400000,\n'
' "ctx" : 323954,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 94.840870,\n'
' "50" : 5.042628,\n'
' "100" : 0.106010,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=86632: Tue '
'Oct 25 23:35:33 2022\n'
' read: IOPS=92.6k, BW=362MiB/s '
'(379MB/s)(2531MiB/7001msec)\n'
' slat (nsec): min=2304, '
'max=769522, avg=2713.92, '
'stdev=997.46\n'
' clat (nsec): min=341, '
'max=775153, avg=18398.81, '
'stdev=3486.35\n'
' lat (usec): min=12, '
'max=792, avg=21.11, stdev= 3.62\n'
' clat percentiles (nsec):\n'
' | 1.00th=[16768], '
'5.00th=[16768], 10.00th=[17024], '
'20.00th=[17280],\n'
' | 30.00th=[17536], '
'40.00th=[17792], '
'50.00th=[17792], '
'60.00th=[18048],\n'
' | 70.00th=[18048], '
'80.00th=[18304], '
'90.00th=[18304], '
'95.00th=[20864],\n'
' | 99.00th=[34560], '
'99.50th=[43776], '
'99.90th=[50432], '
'99.95th=[63232],\n'
' | 99.99th=[75264]\n'
' bw ( KiB/s): min=365080, '
'max=384096, per=100.00%, '
'avg=370353.71, stdev=8054.40, '
'samples=14\n'
' iops : min=91270, '
'max=96024, avg=92588.43, '
'stdev=2013.60, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=94.84%, 50=5.04%, '
'100=0.11%\n'
' lat (usec) : 250=0.01%, '
'1000=0.01%\n'
' cpu : usr=11.17%, '
'sys=40.40%, ctx=323954, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=648055,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=362MiB/s (379MB/s), '
'362MiB/s-362MiB/s '
'(379MB/s-379MB/s), io=2531MiB '
'(2654MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_156.output'),
'state': {'begin': 1666726644.3610137,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7959699630737305,
'end': 1666726645.1569836,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_156.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_156.state')},
'cmd_157': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_157.output'),
'state': {'begin': 1666726646.1621208,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7792618274688721,
'end': 1666726646.9413826,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_157.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_157.state')},
'cmd_158': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_158.output'),
'state': {'begin': 1666726646.9433987,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring-xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.05719804763794,
'end': 1666726658.0005968,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_158.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_158.state')},
'cmd_159': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733747,\n'
' "timestamp_ms" : '
'1666733747490,\n'
' "time" : "Tue Oct 25 23:35:47 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2740174848,\n'
' "io_kbytes" : 2675952,\n'
' "bw_bytes" : 391397635,\n'
' "bw" : 382224,\n'
' "iops" : 95555.920583,\n'
' "runtime" : 7001,\n'
' "total_ios" : 668987,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2274,\n'
' "max" : 30738,\n'
' "mean" : 2583.744357,\n'
' "stddev" : '
'358.833559,\n'
' "N" : 668987\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 430,\n'
' "max" : 156663,\n'
' "mean" : '
'17860.045107,\n'
' "stddev" : '
'3321.104405,\n'
' "N" : 668988,\n'
' "percentile" : {\n'
' "1.000000" : 16768,\n'
' "5.000000" : 16768,\n'
' "10.000000" : '
'16768,\n'
' "20.000000" : '
'17024,\n'
' "30.000000" : '
'17024,\n'
' "40.000000" : '
'17024,\n'
' "50.000000" : '
'17280,\n'
' "60.000000" : '
'17280,\n'
' "70.000000" : '
'17536,\n'
' "80.000000" : '
'17536,\n'
' "90.000000" : '
'17792,\n'
' "95.000000" : '
'19584,\n'
' "99.000000" : '
'34048,\n'
' "99.500000" : '
'40704,\n'
' "99.900000" : '
'51456,\n'
' "99.950000" : '
'64768,\n'
' "99.990000" : 76288\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12112,\n'
' "max" : 159028,\n'
' "mean" : '
'20443.789060,\n'
' "stddev" : '
'3321.208162,\n'
' "N" : 668988\n'
' },\n'
' "bw_min" : 380880,\n'
' "bw_max" : 384376,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'382313.500000,\n'
' "bw_dev" : 1209.400917,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 95220,\n'
' "iops_max" : 96094,\n'
' "iops_mean" : '
'95578.357143,\n'
' "iops_stddev" : '
'302.342548,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.242857,\n'
' "sys_cpu" : 41.428571,\n'
' "ctx" : 334337,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 95.022474,\n'
' "50" : 4.850617,\n'
' "100" : 0.116744,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=86671: Tue '
'Oct 25 23:35:47 2022\n'
' read: IOPS=95.6k, BW=373MiB/s '
'(391MB/s)(2613MiB/7001msec)\n'
' slat (nsec): min=2274, '
'max=30738, avg=2583.74, '
'stdev=358.83\n'
' clat (nsec): min=430, '
'max=156663, avg=17860.05, '
'stdev=3321.10\n'
' lat (usec): min=12, '
'max=159, avg=20.44, stdev= 3.32\n'
' clat percentiles (nsec):\n'
' | 1.00th=[16768], '
'5.00th=[16768], 10.00th=[16768], '
'20.00th=[17024],\n'
' | 30.00th=[17024], '
'40.00th=[17024], '
'50.00th=[17280], '
'60.00th=[17280],\n'
' | 70.00th=[17536], '
'80.00th=[17536], '
'90.00th=[17792], '
'95.00th=[19584],\n'
' | 99.00th=[34048], '
'99.50th=[40704], '
'99.90th=[51456], '
'99.95th=[64768],\n'
' | 99.99th=[76288]\n'
' bw ( KiB/s): min=380880, '
'max=384376, per=100.00%, '
'avg=382313.50, stdev=1209.40, '
'samples=14\n'
' iops : min=95220, '
'max=96094, avg=95578.36, '
'stdev=302.34, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=95.02%, 50=4.85%, '
'100=0.12%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=13.24%, '
'sys=41.43%, ctx=334337, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=668987,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=373MiB/s (391MB/s), '
'373MiB/s-373MiB/s '
'(391MB/s-391MB/s), io=2613MiB '
'(2740MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_159.output'),
'state': {'begin': 1666726658.007309,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7918438911437988,
'end': 1666726658.7991529,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_159.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_159.state')},
'cmd_16': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_16.output'),
'state': {'begin': 1666725969.607921,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.782355785369873,
'end': 1666725970.3902767,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_16.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_16.state')},
'cmd_160': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_io_uring-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_160.output'),
'state': {'begin': 1666726659.7862983,
'cmd': 'rm '
'/tmp/fio-output_4K_2_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7929518222808838,
'end': 1666726660.57925,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_160.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_160.state')},
'cmd_161': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_161.output'),
'state': {'begin': 1666726660.5814507,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_io_uring-xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.05913257598877,
'end': 1666726671.6405833,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_161.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_161.state')},
'cmd_162': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733761,\n'
' "timestamp_ms" : '
'1666733761130,\n'
' "time" : "Tue Oct 25 23:36:01 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2729132032,\n'
' "io_kbytes" : 2665168,\n'
' "bw_bytes" : 389820315,\n'
' "bw" : 380683,\n'
' "iops" : 95170.689901,\n'
' "runtime" : 7001,\n'
' "total_ios" : 666290,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2294,\n'
' "max" : 39635,\n'
' "mean" : 2615.013979,\n'
' "stddev" : '
'376.495980,\n'
' "N" : 666291\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 370,\n'
' "max" : 191489,\n'
' "mean" : '
'17882.856067,\n'
' "stddev" : '
'3375.558001,\n'
' "N" : 666291,\n'
' "percentile" : {\n'
' "1.000000" : 16768,\n'
' "5.000000" : 16768,\n'
' "10.000000" : '
'16768,\n'
' "20.000000" : '
'16768,\n'
' "30.000000" : '
'17024,\n'
' "40.000000" : '
'17024,\n'
' "50.000000" : '
'17280,\n'
' "60.000000" : '
'17536,\n'
' "70.000000" : '
'17536,\n'
' "80.000000" : '
'17536,\n'
' "90.000000" : '
'17792,\n'
' "95.000000" : '
'19584,\n'
' "99.000000" : '
'34048,\n'
' "99.500000" : '
'43264,\n'
' "99.900000" : '
'53504,\n'
' "99.950000" : '
'65280,\n'
' "99.990000" : 76288\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11221,\n'
' "max" : 193854,\n'
' "mean" : '
'20497.870046,\n'
' "stddev" : '
'3378.217884,\n'
' "N" : 666291\n'
' },\n'
' "bw_min" : 378240,\n'
' "bw_max" : 382600,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'380772.071429,\n'
' "bw_dev" : 1090.263798,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 94560,\n'
' "iops_max" : 95650,\n'
' "iops_mean" : '
'95193.000000,\n'
' "iops_stddev" : '
'272.562484,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.257143,\n'
' "sys_cpu" : 39.942857,\n'
' "ctx" : 332996,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 95.048703,\n'
' "50" : 4.824326,\n'
' "100" : 0.118717,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=86711: Tue '
'Oct 25 23:36:01 2022\n'
' read: IOPS=95.2k, BW=372MiB/s '
'(390MB/s)(2603MiB/7001msec)\n'
' slat (nsec): min=2294, '
'max=39635, avg=2615.01, '
'stdev=376.50\n'
' clat (nsec): min=370, '
'max=191489, avg=17882.86, '
'stdev=3375.56\n'
' lat (usec): min=11, '
'max=193, avg=20.50, stdev= 3.38\n'
' clat percentiles (nsec):\n'
' | 1.00th=[16768], '
'5.00th=[16768], 10.00th=[16768], '
'20.00th=[16768],\n'
' | 30.00th=[17024], '
'40.00th=[17024], '
'50.00th=[17280], '
'60.00th=[17536],\n'
' | 70.00th=[17536], '
'80.00th=[17536], '
'90.00th=[17792], '
'95.00th=[19584],\n'
' | 99.00th=[34048], '
'99.50th=[43264], '
'99.90th=[53504], '
'99.95th=[65280],\n'
' | 99.99th=[76288]\n'
' bw ( KiB/s): min=378240, '
'max=382600, per=100.00%, '
'avg=380772.07, stdev=1090.26, '
'samples=14\n'
' iops : min=94560, '
'max=95650, avg=95193.00, '
'stdev=272.56, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=95.05%, 50=4.82%, '
'100=0.12%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=15.26%, '
'sys=39.94%, ctx=332996, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=666290,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=372MiB/s (390MB/s), '
'372MiB/s-372MiB/s '
'(390MB/s-390MB/s), io=2603MiB '
'(2729MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_162.output'),
'state': {'begin': 1666726671.643497,
'cmd': 'cat '
'/tmp/fio-output_4K_2_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7917177677154541,
'end': 1666726672.4352148,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_162.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_162.state')},
'cmd_163': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_libaio-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_163.output'),
'state': {'begin': 1666726673.4310272,
'cmd': 'rm '
'/tmp/fio-output_4K_2_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7887732982635498,
'end': 1666726674.2198005,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_163.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_163.state')},
'cmd_164': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_164.output'),
'state': {'begin': 1666726674.2219372,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_libaio-reference_0.txt"',
'cwd': 'None',
'elapsed': 11.08357286453247,
'end': 1666726685.30551,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_164.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_164.state')},
'cmd_165': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=libaio, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733774,\n'
' "timestamp_ms" : '
'1666733774795,\n'
' "time" : "Tue Oct 25 23:36:14 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2756984832,\n'
' "io_kbytes" : 2692368,\n'
' "bw_bytes" : 393798719,\n'
' "bw" : 384569,\n'
' "iops" : 96142.122554,\n'
' "runtime" : 7001,\n'
' "total_ios" : 673091,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2394,\n'
' "max" : 29245,\n'
' "mean" : 2705.547284,\n'
' "stddev" : '
'335.582093,\n'
' "N" : 673091\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 802,\n'
' "max" : 199354,\n'
' "mean" : '
'17555.780315,\n'
' "stddev" : '
'5106.894427,\n'
' "N" : 673092,\n'
' "percentile" : {\n'
' "1.000000" : 12736,\n'
' "5.000000" : 12864,\n'
' "10.000000" : '
'12864,\n'
' "20.000000" : '
'12864,\n'
' "30.000000" : '
'12992,\n'
' "40.000000" : '
'13120,\n'
' "50.000000" : '
'20608,\n'
' "60.000000" : '
'20864,\n'
' "70.000000" : '
'20864,\n'
' "80.000000" : '
'20864,\n'
' "90.000000" : '
'21120,\n'
' "95.000000" : '
'21632,\n'
' "99.000000" : '
'33024,\n'
' "99.500000" : '
'38656,\n'
' "99.900000" : '
'48896,\n'
' "99.950000" : '
'61696,\n'
' "99.990000" : 76288\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11712,\n'
' "max" : 202269,\n'
' "mean" : '
'20261.327762,\n'
' "stddev" : '
'5287.935425,\n'
' "N" : 673092\n'
' },\n'
' "bw_min" : 383832,\n'
' "bw_max" : 385720,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'384720.071429,\n'
' "bw_dev" : 610.691722,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 95958,\n'
' "iops_max" : 96430,\n'
' "iops_mean" : '
'96179.928571,\n'
' "iops_stddev" : '
'152.704070,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.714286,\n'
' "sys_cpu" : 42.514286,\n'
' "ctx" : 336367,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 47.780909,\n'
' "50" : 52.115241,\n'
' "100" : 0.093895,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 963565,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 11496,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 11496,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=86751: Tue '
'Oct 25 23:36:14 2022\n'
' read: IOPS=96.1k, BW=376MiB/s '
'(394MB/s)(2629MiB/7001msec)\n'
' slat (nsec): min=2394, '
'max=29245, avg=2705.55, '
'stdev=335.58\n'
' clat (nsec): min=802, '
'max=199354, avg=17555.78, '
'stdev=5106.89\n'
' lat (usec): min=11, '
'max=202, avg=20.26, stdev= 5.29\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12736], '
'5.00th=[12864], 10.00th=[12864], '
'20.00th=[12864],\n'
' | 30.00th=[12992], '
'40.00th=[13120], '
'50.00th=[20608], '
'60.00th=[20864],\n'
' | 70.00th=[20864], '
'80.00th=[20864], '
'90.00th=[21120], '
'95.00th=[21632],\n'
' | 99.00th=[33024], '
'99.50th=[38656], '
'99.90th=[48896], '
'99.95th=[61696],\n'
' | 99.99th=[76288]\n'
' bw ( KiB/s): min=383832, '
'max=385720, per=100.00%, '
'avg=384720.07, stdev=610.69, '
'samples=14\n'
' iops : min=95958, '
'max=96430, avg=96179.93, '
'stdev=152.70, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=47.78%, 50=52.12%, '
'100=0.09%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=14.71%, '
'sys=42.51%, ctx=336367, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=673091,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=376MiB/s (394MB/s), '
'376MiB/s-376MiB/s '
'(394MB/s-394MB/s), io=2629MiB '
'(2757MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=963565/0, '
'merge=0/0, ticks=11496/0, '
'in_queue=11496, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_165.output'),
'state': {'begin': 1666726685.308098,
'cmd': 'cat '
'/tmp/fio-output_4K_2_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7916755676269531,
'end': 1666726686.0997736,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_165.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_165.state')},
'cmd_166': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_libaio-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_166.output'),
'state': {'begin': 1666726687.1028388,
'cmd': 'rm '
'/tmp/fio-output_4K_2_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7810490131378174,
'end': 1666726687.8838878,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_166.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_166.state')},
'cmd_167': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_167.output'),
'state': {'begin': 1666726687.8861325,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_libaio-reference_1.txt"',
'cwd': 'None',
'elapsed': 11.079588413238525,
'end': 1666726698.965721,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_167.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_167.state')},
'cmd_168': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=libaio, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733788,\n'
' "timestamp_ms" : '
'1666733788456,\n'
' "time" : "Tue Oct 25 23:36:28 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2764947456,\n'
' "io_kbytes" : 2700144,\n'
' "bw_bytes" : 394936074,\n'
' "bw" : 385679,\n'
' "iops" : 96419.654335,\n'
' "runtime" : 7001,\n'
' "total_ios" : 675034,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2384,\n'
' "max" : 46998,\n'
' "mean" : 2698.317754,\n'
' "stddev" : '
'340.659642,\n'
' "N" : 675035\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 882,\n'
' "max" : 136166,\n'
' "mean" : '
'17524.151786,\n'
' "stddev" : '
'5078.250149,\n'
' "N" : 675035,\n'
' "percentile" : {\n'
' "1.000000" : 12736,\n'
' "5.000000" : 12864,\n'
' "10.000000" : '
'12864,\n'
' "20.000000" : '
'12992,\n'
' "30.000000" : '
'12992,\n'
' "40.000000" : '
'13120,\n'
' "50.000000" : '
'20608,\n'
' "60.000000" : '
'20608,\n'
' "70.000000" : '
'20864,\n'
' "80.000000" : '
'20864,\n'
' "90.000000" : '
'20864,\n'
' "95.000000" : '
'21632,\n'
' "99.000000" : '
'33024,\n'
' "99.500000" : '
'39168,\n'
' "99.900000" : '
'52480,\n'
' "99.950000" : '
'66048,\n'
' "99.990000" : 77312\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12653,\n'
' "max" : 139051,\n'
' "mean" : '
'20222.470281,\n'
' "stddev" : '
'5270.579236,\n'
' "N" : 675035\n'
' },\n'
' "bw_min" : 385120,\n'
' "bw_max" : 386536,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'385827.285714,\n'
' "bw_dev" : 381.399332,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 96280,\n'
' "iops_max" : 96634,\n'
' "iops_mean" : '
'96456.714286,\n'
' "iops_stddev" : '
'95.320051,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.000000,\n'
' "sys_cpu" : 43.514286,\n'
' "ctx" : 337349,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 47.800703,\n'
' "50" : 52.086265,\n'
' "100" : 0.105328,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 965663,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 11565,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 11565,\n'
' "util" : 99.049128\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=86789: Tue '
'Oct 25 23:36:28 2022\n'
' read: IOPS=96.4k, BW=377MiB/s '
'(395MB/s)(2637MiB/7001msec)\n'
' slat (nsec): min=2384, '
'max=46998, avg=2698.32, '
'stdev=340.66\n'
' clat (nsec): min=882, '
'max=136166, avg=17524.15, '
'stdev=5078.25\n'
' lat (usec): min=12, '
'max=139, avg=20.22, stdev= 5.27\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12736], '
'5.00th=[12864], 10.00th=[12864], '
'20.00th=[12992],\n'
' | 30.00th=[12992], '
'40.00th=[13120], '
'50.00th=[20608], '
'60.00th=[20608],\n'
' | 70.00th=[20864], '
'80.00th=[20864], '
'90.00th=[20864], '
'95.00th=[21632],\n'
' | 99.00th=[33024], '
'99.50th=[39168], '
'99.90th=[52480], '
'99.95th=[66048],\n'
' | 99.99th=[77312]\n'
' bw ( KiB/s): min=385120, '
'max=386536, per=100.00%, '
'avg=385827.29, stdev=381.40, '
'samples=14\n'
' iops : min=96280, '
'max=96634, avg=96456.71, '
'stdev=95.32, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=47.80%, 50=52.09%, '
'100=0.11%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=13.00%, '
'sys=43.51%, ctx=337349, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=675034,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=377MiB/s (395MB/s), '
'377MiB/s-377MiB/s '
'(395MB/s-395MB/s), io=2637MiB '
'(2765MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=965663/0, '
'merge=0/0, ticks=11565/0, '
'in_queue=11565, util=99.05%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_168.output'),
'state': {'begin': 1666726698.9724402,
'cmd': 'cat '
'/tmp/fio-output_4K_2_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7907559871673584,
'end': 1666726699.7631962,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_168.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_168.state')},
'cmd_169': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_libaio-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_169.output'),
'state': {'begin': 1666726700.7536056,
'cmd': 'rm '
'/tmp/fio-output_4K_2_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7884535789489746,
'end': 1666726701.5420592,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_169.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_169.state')},
'cmd_17': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_17.output'),
'state': {'begin': 1666725970.3925776,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.057618141174316,
'end': 1666725981.4501958,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_17.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_17.state')},
'cmd_170': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_170.output'),
'state': {'begin': 1666726701.543887,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_libaio-reference_2.txt"',
'cwd': 'None',
'elapsed': 11.06456971168518,
'end': 1666726712.6084566,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_170.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_170.state')},
'cmd_171': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=libaio, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733802,\n'
' "timestamp_ms" : '
'1666733802098,\n'
' "time" : "Tue Oct 25 23:36:42 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2776817664,\n'
' "io_kbytes" : 2711736,\n'
' "bw_bytes" : 396631576,\n'
' "bw" : 387335,\n'
' "iops" : 96833.595201,\n'
' "runtime" : 7001,\n'
' "total_ios" : 677932,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2374,\n'
' "max" : 177503,\n'
' "mean" : 2666.347233,\n'
' "stddev" : '
'383.992040,\n'
' "N" : 677933\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 822,\n'
' "max" : 195737,\n'
' "mean" : '
'17461.917285,\n'
' "stddev" : '
'5079.205885,\n'
' "N" : 677933,\n'
' "percentile" : {\n'
' "1.000000" : 12736,\n'
' "5.000000" : 12736,\n'
' "10.000000" : '
'12864,\n'
' "20.000000" : '
'12864,\n'
' "30.000000" : '
'12992,\n'
' "40.000000" : '
'13120,\n'
' "50.000000" : '
'20352,\n'
' "60.000000" : '
'20608,\n'
' "70.000000" : '
'20608,\n'
' "80.000000" : '
'20864,\n'
' "90.000000" : '
'20864,\n'
' "95.000000" : '
'21632,\n'
' "99.000000" : '
'33024,\n'
' "99.500000" : '
'38144,\n'
' "99.900000" : '
'50944,\n'
' "99.950000" : '
'65280,\n'
' "99.990000" : 77312\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 13045,\n'
' "max" : 198572,\n'
' "mean" : '
'20128.265079,\n'
' "stddev" : '
'5254.394288,\n'
' "N" : 677933\n'
' },\n'
' "bw_min" : 385704,\n'
' "bw_max" : 389835,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'387489.428571,\n'
' "bw_dev" : 1265.535503,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 96426,\n'
' "iops_max" : 97458,\n'
' "iops_mean" : '
'96872.285714,\n'
' "iops_stddev" : '
'316.254870,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.857143,\n'
' "sys_cpu" : 41.557143,\n'
' "ctx" : 338781,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 47.810990,\n'
' "50" : 52.078527,\n'
' "100" : 0.101043,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 970953,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 11596,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 11596,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=86827: Tue '
'Oct 25 23:36:42 2022\n'
' read: IOPS=96.8k, BW=378MiB/s '
'(397MB/s)(2648MiB/7001msec)\n'
' slat (nsec): min=2374, '
'max=177503, avg=2666.35, '
'stdev=383.99\n'
' clat (nsec): min=822, '
'max=195737, avg=17461.92, '
'stdev=5079.21\n'
' lat (usec): min=13, '
'max=198, avg=20.13, stdev= 5.25\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12736], '
'5.00th=[12736], 10.00th=[12864], '
'20.00th=[12864],\n'
' | 30.00th=[12992], '
'40.00th=[13120], '
'50.00th=[20352], '
'60.00th=[20608],\n'
' | 70.00th=[20608], '
'80.00th=[20864], '
'90.00th=[20864], '
'95.00th=[21632],\n'
' | 99.00th=[33024], '
'99.50th=[38144], '
'99.90th=[50944], '
'99.95th=[65280],\n'
' | 99.99th=[77312]\n'
' bw ( KiB/s): min=385704, '
'max=389835, per=100.00%, '
'avg=387489.43, stdev=1265.54, '
'samples=14\n'
' iops : min=96426, '
'max=97458, avg=96872.29, '
'stdev=316.25, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 10=0.01%, '
'20=47.81%, 50=52.08%, 100=0.10%, '
'250=0.01%\n'
' cpu : usr=14.86%, '
'sys=41.56%, ctx=338781, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=677932,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=378MiB/s (397MB/s), '
'378MiB/s-378MiB/s '
'(397MB/s-397MB/s), io=2648MiB '
'(2777MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=970953/0, '
'merge=0/0, ticks=11596/0, '
'in_queue=11596, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_171.output'),
'state': {'begin': 1666726712.611358,
'cmd': 'cat '
'/tmp/fio-output_4K_2_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7903308868408203,
'end': 1666726713.4016888,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_171.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_171.state')},
'cmd_172': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_libaio-bdev_aio_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_172.output'),
'state': {'begin': 1666726715.4345553,
'cmd': 'rm '
'/tmp/fio-output_4K_2_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.7796828746795654,
'end': 1666726716.2142382,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_172.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_172.state')},
'cmd_173': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_173.output'),
'state': {'begin': 1666726716.2163448,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_libaio-bdev_aio_0.txt"',
'cwd': 'None',
'elapsed': 11.510746002197266,
'end': 1666726727.7270908,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_173.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_173.state')},
'cmd_174': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733817,\n'
' "timestamp_ms" : '
'1666733817120,\n'
' "time" : "Tue Oct 25 23:36:57 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3817984000,\n'
' "io_kbytes" : 3728500,\n'
' "bw_bytes" : 545348378,\n'
' "bw" : 532566,\n'
' "iops" : 133141.551207,\n'
' "runtime" : 7001,\n'
' "total_ios" : 932124,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1503,\n'
' "max" : 15279,\n'
' "mean" : 1580.962068,\n'
' "stddev" : '
'125.314578,\n'
' "N" : 932125\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 261,\n'
' "max" : 116468,\n'
' "mean" : '
'12250.310121,\n'
' "stddev" : '
'3714.120484,\n'
' "N" : 932125,\n'
' "percentile" : {\n'
' "1.000000" : 10560,\n'
' "5.000000" : 10560,\n'
' "10.000000" : '
'10560,\n'
' "20.000000" : '
'10688,\n'
' "30.000000" : '
'10688,\n'
' "40.000000" : '
'10816,\n'
' "50.000000" : '
'12224,\n'
' "60.000000" : '
'12352,\n'
' "70.000000" : '
'12352,\n'
' "80.000000" : '
'12480,\n'
' "90.000000" : '
'12608,\n'
' "95.000000" : '
'13120,\n'
' "99.000000" : '
'29568,\n'
' "99.500000" : '
'37120,\n'
' "99.900000" : '
'45312,\n'
' "99.950000" : '
'63232,\n'
' "99.990000" : 74240\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9247,\n'
' "max" : 118081,\n'
' "mean" : '
'13831.272190,\n'
' "stddev" : '
'3720.542513,\n'
' "N" : 932125\n'
' },\n'
' "bw_min" : 531104,\n'
' "bw_max" : 534640,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'532689.428571,\n'
' "bw_dev" : 1101.338606,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 132776,\n'
' "iops_max" : 133660,\n'
' "iops_mean" : '
'133172.214286,\n'
' "iops_stddev" : '
'275.363306,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 84.400000,\n'
' "sys_cpu" : 15.585714,\n'
' "ctx" : 4,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 49.981869,\n'
' "2" : 50.018131,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.049886,\n'
' "20" : 96.550674,\n'
' "50" : 3.300312,\n'
' "100" : 0.096232,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=86875: Tue '
'Oct 25 23:36:57 2022\n'
' read: IOPS=133k, BW=520MiB/s '
'(545MB/s)(3641MiB/7001msec)\n'
' slat (nsec): min=1503, '
'max=15279, avg=1580.96, '
'stdev=125.31\n'
' clat (nsec): min=261, '
'max=116468, avg=12250.31, '
'stdev=3714.12\n'
' lat (usec): min=9, max=118, '
'avg=13.83, stdev= 3.72\n'
' clat percentiles (nsec):\n'
' | 1.00th=[10560], '
'5.00th=[10560], 10.00th=[10560], '
'20.00th=[10688],\n'
' | 30.00th=[10688], '
'40.00th=[10816], '
'50.00th=[12224], '
'60.00th=[12352],\n'
' | 70.00th=[12352], '
'80.00th=[12480], '
'90.00th=[12608], '
'95.00th=[13120],\n'
' | 99.00th=[29568], '
'99.50th=[37120], '
'99.90th=[45312], '
'99.95th=[63232],\n'
' | 99.99th=[74240]\n'
' bw ( KiB/s): min=531104, '
'max=534640, per=100.00%, '
'avg=532689.43, stdev=1101.34, '
'samples=14\n'
' iops : min=132776, '
'max=133660, avg=133172.21, '
'stdev=275.36, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.05%, 20=96.55%, 50=3.30%, '
'100=0.10%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=84.40%, '
'sys=15.59%, ctx=4, majf=0, '
'minf=0\n'
' IO depths : 1=50.0%, '
'2=50.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=932124,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=520MiB/s (545MB/s), '
'520MiB/s-520MiB/s '
'(545MB/s-545MB/s), io=3641MiB '
'(3818MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_174.output'),
'state': {'begin': 1666726727.7298813,
'cmd': 'cat '
'/tmp/fio-output_4K_2_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.7981071472167969,
'end': 1666726728.5279884,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_174.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_174.state')},
'cmd_175': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_libaio-bdev_aio_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_175.output'),
'state': {'begin': 1666726730.5578527,
'cmd': 'rm '
'/tmp/fio-output_4K_2_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7900984287261963,
'end': 1666726731.3479512,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_175.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_175.state')},
'cmd_176': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_176.output'),
'state': {'begin': 1666726731.3500626,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_libaio-bdev_aio_1.txt"',
'cwd': 'None',
'elapsed': 11.429170370101929,
'end': 1666726742.779233,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_176.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_176.state')},
'cmd_177': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733832,\n'
' "timestamp_ms" : '
'1666733832158,\n'
' "time" : "Tue Oct 25 23:37:12 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3812896768,\n'
' "io_kbytes" : 3723532,\n'
' "bw_bytes" : 544621735,\n'
' "bw" : 531857,\n'
' "iops" : 132964.147979,\n'
' "runtime" : 7001,\n'
' "total_ios" : 930882,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1542,\n'
' "max" : 43872,\n'
' "mean" : 1620.137457,\n'
' "stddev" : '
'131.643678,\n'
' "N" : 930882\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 240,\n'
' "max" : 1034130,\n'
' "mean" : '
'12198.356105,\n'
' "stddev" : '
'4029.463553,\n'
' "N" : 930882,\n'
' "percentile" : {\n'
' "1.000000" : 10432,\n'
' "5.000000" : 10432,\n'
' "10.000000" : '
'10560,\n'
' "20.000000" : '
'10560,\n'
' "30.000000" : '
'10688,\n'
' "40.000000" : '
'10688,\n'
' "50.000000" : '
'12224,\n'
' "60.000000" : '
'12352,\n'
' "70.000000" : '
'12352,\n'
' "80.000000" : '
'12480,\n'
' "90.000000" : '
'12480,\n'
' "95.000000" : '
'13120,\n'
' "99.000000" : '
'29568,\n'
' "99.500000" : '
'37120,\n'
' "99.900000" : '
'50432,\n'
' "99.950000" : '
'63744,\n'
' "99.990000" : 74240\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9217,\n'
' "max" : 1035763,\n'
' "mean" : '
'13818.493562,\n'
' "stddev" : '
'4036.311352,\n'
' "N" : 930882\n'
' },\n'
' "bw_min" : 528936,\n'
' "bw_max" : 533304,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'531980.714286,\n'
' "bw_dev" : 1458.161879,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 132234,\n'
' "iops_max" : 133326,\n'
' "iops_mean" : '
'132995.142857,\n'
' "iops_stddev" : '
'364.505747,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 86.657143,\n'
' "sys_cpu" : 13.300000,\n'
' "ctx" : 41,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 49.984746,\n'
' "2" : 50.015254,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.049738,\n'
' "20" : 96.555847,\n'
' "50" : 3.289031,\n'
' "100" : 0.102483,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=86924: Tue '
'Oct 25 23:37:12 2022\n'
' read: IOPS=133k, BW=519MiB/s '
'(545MB/s)(3636MiB/7001msec)\n'
' slat (nsec): min=1542, '
'max=43872, avg=1620.14, '
'stdev=131.64\n'
' clat (nsec): min=240, '
'max=1034.1k, avg=12198.36, '
'stdev=4029.46\n'
' lat (usec): min=9, '
'max=1035, avg=13.82, stdev= '
'4.04\n'
' clat percentiles (nsec):\n'
' | 1.00th=[10432], '
'5.00th=[10432], 10.00th=[10560], '
'20.00th=[10560],\n'
' | 30.00th=[10688], '
'40.00th=[10688], '
'50.00th=[12224], '
'60.00th=[12352],\n'
' | 70.00th=[12352], '
'80.00th=[12480], '
'90.00th=[12480], '
'95.00th=[13120],\n'
' | 99.00th=[29568], '
'99.50th=[37120], '
'99.90th=[50432], '
'99.95th=[63744],\n'
' | 99.99th=[74240]\n'
' bw ( KiB/s): min=528936, '
'max=533304, per=100.00%, '
'avg=531980.71, stdev=1458.16, '
'samples=14\n'
' iops : min=132234, '
'max=133326, avg=132995.14, '
'stdev=364.51, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.05%, 20=96.56%, 50=3.29%, '
'100=0.10%\n'
' lat (usec) : 250=0.01%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=86.66%, '
'sys=13.30%, ctx=41, majf=0, '
'minf=0\n'
' IO depths : 1=50.0%, '
'2=50.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=930882,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=519MiB/s (545MB/s), '
'519MiB/s-519MiB/s '
'(545MB/s-545MB/s), io=3636MiB '
'(3813MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_177.output'),
'state': {'begin': 1666726742.781971,
'cmd': 'cat '
'/tmp/fio-output_4K_2_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7942378520965576,
'end': 1666726743.5762088,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_177.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_177.state')},
'cmd_178': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_libaio-bdev_aio_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_178.output'),
'state': {'begin': 1666726745.5864446,
'cmd': 'rm '
'/tmp/fio-output_4K_2_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.7919878959655762,
'end': 1666726746.3784325,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_178.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_178.state')},
'cmd_179': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_179.output'),
'state': {'begin': 1666726746.380755,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_libaio-bdev_aio_2.txt"',
'cwd': 'None',
'elapsed': 11.52251935005188,
'end': 1666726757.9032743,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_179.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_179.state')},
'cmd_18': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733070,\n'
' "timestamp_ms" : '
'1666733070951,\n'
' "time" : "Tue Oct 25 23:24:30 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1847742464,\n'
' "io_kbytes" : 1804436,\n'
' "bw_bytes" : 263925505,\n'
' "bw" : 257739,\n'
' "iops" : 64434.937866,\n'
' "runtime" : 7001,\n'
' "total_ios" : 451109,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2314,\n'
' "max" : 18234,\n'
' "mean" : 2506.600283,\n'
' "stddev" : 209.404360,\n'
' "N" : 451109\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 341,\n'
' "max" : 183043,\n'
' "mean" : 12344.901031,\n'
' "stddev" : '
'2052.638325,\n'
' "N" : 451109,\n'
' "percentile" : {\n'
' "1.000000" : 11840,\n'
' "5.000000" : 11840,\n'
' "10.000000" : 11840,\n'
' "20.000000" : 11968,\n'
' "30.000000" : 11968,\n'
' "40.000000" : 11968,\n'
' "50.000000" : 12096,\n'
' "60.000000" : 12096,\n'
' "70.000000" : 12224,\n'
' "80.000000" : 12224,\n'
' "90.000000" : 12352,\n'
' "95.000000" : 12480,\n'
' "99.000000" : 22144,\n'
' "99.500000" : 24448,\n'
' "99.900000" : 31616,\n'
' "99.950000" : 49920,\n'
' "99.990000" : 64256\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11482,\n'
' "max" : 185648,\n'
' "mean" : 14851.501313,\n'
' "stddev" : '
'2066.668313,\n'
' "N" : 451109\n'
' },\n'
' "bw_min" : 257312,\n'
' "bw_max" : 258252,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'257799.142857,\n'
' "bw_dev" : 320.139118,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 64328,\n'
' "iops_max" : 64563,\n'
' "iops_mean" : '
'64449.785714,\n'
' "iops_stddev" : '
'80.034780,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 10.671429,\n'
' "sys_cpu" : 36.157143,\n'
' "ctx" : 451088,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 98.461569,\n'
' "50" : 1.477913,\n'
' "100" : 0.051650,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=84584: Tue '
'Oct 25 23:24:30 2022\n'
' read: IOPS=64.4k, BW=252MiB/s '
'(264MB/s)(1762MiB/7001msec)\n'
' slat (nsec): min=2314, '
'max=18234, avg=2506.60, '
'stdev=209.40\n'
' clat (nsec): min=341, '
'max=183043, avg=12344.90, '
'stdev=2052.64\n'
' lat (usec): min=11, max=185, '
'avg=14.85, stdev= 2.07\n'
' clat percentiles (nsec):\n'
' | 1.00th=[11840], '
'5.00th=[11840], 10.00th=[11840], '
'20.00th=[11968],\n'
' | 30.00th=[11968], '
'40.00th=[11968], 50.00th=[12096], '
'60.00th=[12096],\n'
' | 70.00th=[12224], '
'80.00th=[12224], 90.00th=[12352], '
'95.00th=[12480],\n'
' | 99.00th=[22144], '
'99.50th=[24448], 99.90th=[31616], '
'99.95th=[49920],\n'
' | 99.99th=[64256]\n'
' bw ( KiB/s): min=257312, '
'max=258252, per=100.00%, '
'avg=257799.14, stdev=320.14, '
'samples=14\n'
' iops : min=64328, '
'max=64563, avg=64449.79, '
'stdev=80.03, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=98.46%, 50=1.48%, '
'100=0.05%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=10.67%, '
'sys=36.16%, ctx=451088, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=451109,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=252MiB/s (264MB/s), '
'252MiB/s-252MiB/s '
'(264MB/s-264MB/s), io=1762MiB '
'(1848MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_18.output'),
'state': {'begin': 1666725981.4536474,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7915840148925781,
'end': 1666725982.2452314,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_18.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_18.state')},
'cmd_180': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733847,\n'
' "timestamp_ms" : '
'1666733847292,\n'
' "time" : "Tue Oct 25 23:37:27 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3812159488,\n'
' "io_kbytes" : 3722812,\n'
' "bw_bytes" : 544516424,\n'
' "bw" : 531754,\n'
' "iops" : 132938.437366,\n'
' "runtime" : 7001,\n'
' "total_ios" : 930702,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1542,\n'
' "max" : 16170,\n'
' "mean" : 1619.645983,\n'
' "stddev" : '
'121.460440,\n'
' "N" : 930702\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 251,\n'
' "max" : 176772,\n'
' "mean" : '
'12194.889915,\n'
' "stddev" : '
'3779.824468,\n'
' "N" : 930702,\n'
' "percentile" : {\n'
' "1.000000" : 10432,\n'
' "5.000000" : 10560,\n'
' "10.000000" : '
'10560,\n'
' "20.000000" : '
'10560,\n'
' "30.000000" : '
'10688,\n'
' "40.000000" : '
'10688,\n'
' "50.000000" : '
'12224,\n'
' "60.000000" : '
'12352,\n'
' "70.000000" : '
'12352,\n'
' "80.000000" : '
'12480,\n'
' "90.000000" : '
'12480,\n'
' "95.000000" : '
'12992,\n'
' "99.000000" : '
'29568,\n'
' "99.500000" : '
'37120,\n'
' "99.900000" : '
'50432,\n'
' "99.950000" : '
'63744,\n'
' "99.990000" : 74240\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9257,\n'
' "max" : 178395,\n'
' "mean" : '
'13814.535899,\n'
' "stddev" : '
'3787.101803,\n'
' "N" : 930702\n'
' },\n'
' "bw_min" : 530488,\n'
' "bw_max" : 532609,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'531886.928571,\n'
' "bw_dev" : 581.030446,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 132622,\n'
' "iops_max" : 133152,\n'
' "iops_mean" : '
'132971.714286,\n'
' "iops_stddev" : '
'145.339617,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 84.900000,\n'
' "sys_cpu" : 15.071429,\n'
' "ctx" : 16,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 49.984635,\n'
' "2" : 50.015365,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.048458,\n'
' "20" : 96.556255,\n'
' "50" : 3.289452,\n'
' "100" : 0.102288,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=86975: Tue '
'Oct 25 23:37:27 2022\n'
' read: IOPS=133k, BW=519MiB/s '
'(545MB/s)(3636MiB/7001msec)\n'
' slat (nsec): min=1542, '
'max=16170, avg=1619.65, '
'stdev=121.46\n'
' clat (nsec): min=251, '
'max=176772, avg=12194.89, '
'stdev=3779.82\n'
' lat (usec): min=9, max=178, '
'avg=13.81, stdev= 3.79\n'
' clat percentiles (nsec):\n'
' | 1.00th=[10432], '
'5.00th=[10560], 10.00th=[10560], '
'20.00th=[10560],\n'
' | 30.00th=[10688], '
'40.00th=[10688], '
'50.00th=[12224], '
'60.00th=[12352],\n'
' | 70.00th=[12352], '
'80.00th=[12480], '
'90.00th=[12480], '
'95.00th=[12992],\n'
' | 99.00th=[29568], '
'99.50th=[37120], '
'99.90th=[50432], '
'99.95th=[63744],\n'
' | 99.99th=[74240]\n'
' bw ( KiB/s): min=530488, '
'max=532609, per=100.00%, '
'avg=531886.93, stdev=581.03, '
'samples=14\n'
' iops : min=132622, '
'max=133152, avg=132971.71, '
'stdev=145.34, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.05%, 20=96.56%, 50=3.29%, '
'100=0.10%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=84.90%, '
'sys=15.07%, ctx=16, majf=0, '
'minf=0\n'
' IO depths : 1=50.0%, '
'2=50.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=930702,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=519MiB/s (545MB/s), '
'519MiB/s-519MiB/s '
'(545MB/s-545MB/s), io=3636MiB '
'(3812MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_180.output'),
'state': {'begin': 1666726757.9060128,
'cmd': 'cat '
'/tmp/fio-output_4K_2_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.7953801155090332,
'end': 1666726758.701393,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_180.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_180.state')},
'cmd_181': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_181.output'),
'state': {'begin': 1666726760.7446208,
'cmd': 'rm '
'/tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7837724685668945,
'end': 1666726761.5283933,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_181.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_181.state')},
'cmd_182': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_182.output'),
'state': {'begin': 1666726761.5305362,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.477898359298706,
'end': 1666726773.0084345,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_182.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_182.state')},
'cmd_183': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733862,\n'
' "timestamp_ms" : '
'1666733862396,\n'
' "time" : "Tue Oct 25 23:37:42 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3758456832,\n'
' "io_kbytes" : 3670368,\n'
' "bw_bytes" : 536845712,\n'
' "bw" : 524263,\n'
' "iops" : 131065.704899,\n'
' "runtime" : 7001,\n'
' "total_ios" : 917591,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1542,\n'
' "max" : 1290761,\n'
' "mean" : 1616.681889,\n'
' "stddev" : '
'1355.420376,\n'
' "N" : 917592\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 691,\n'
' "max" : 1291843,\n'
' "mean" : '
'12424.666600,\n'
' "stddev" : '
'3999.828740,\n'
' "N" : 917592,\n'
' "percentile" : {\n'
' "1.000000" : 10560,\n'
' "5.000000" : 10560,\n'
' "10.000000" : '
'10688,\n'
' "20.000000" : '
'10816,\n'
' "30.000000" : '
'10944,\n'
' "40.000000" : '
'10944,\n'
' "50.000000" : '
'12352,\n'
' "60.000000" : '
'12480,\n'
' "70.000000" : '
'12608,\n'
' "80.000000" : '
'12736,\n'
' "90.000000" : '
'12736,\n'
' "95.000000" : '
'13376,\n'
' "99.000000" : '
'29824,\n'
' "99.500000" : '
'37120,\n'
' "99.900000" : '
'50944,\n'
' "99.950000" : '
'63744,\n'
' "99.990000" : 75264\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9447,\n'
' "max" : 1294137,\n'
' "mean" : '
'14041.348489,\n'
' "stddev" : '
'4225.469205,\n'
' "N" : 917592\n'
' },\n'
' "bw_min" : 523344,\n'
' "bw_max" : 524897,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'524383.500000,\n'
' "bw_dev" : 486.058995,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 130836,\n'
' "iops_max" : 131224,\n'
' "iops_mean" : '
'131095.857143,\n'
' "iops_stddev" : '
'121.494449,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 32.271429,\n'
' "sys_cpu" : 67.685714,\n'
' "ctx" : 43,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 49.985451,\n'
' "2" : 50.014549,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.042394,\n'
' "20" : 96.514896,\n'
' "50" : 3.329261,\n'
' "100" : 0.110616,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=87024: Tue '
'Oct 25 23:37:42 2022\n'
' read: IOPS=131k, BW=512MiB/s '
'(537MB/s)(3584MiB/7001msec)\n'
' slat (nsec): min=1542, '
'max=1290.8k, avg=1616.68, '
'stdev=1355.42\n'
' clat (nsec): min=691, '
'max=1291.8k, avg=12424.67, '
'stdev=3999.83\n'
' lat (usec): min=9, '
'max=1294, avg=14.04, stdev= '
'4.23\n'
' clat percentiles (nsec):\n'
' | 1.00th=[10560], '
'5.00th=[10560], 10.00th=[10688], '
'20.00th=[10816],\n'
' | 30.00th=[10944], '
'40.00th=[10944], '
'50.00th=[12352], '
'60.00th=[12480],\n'
' | 70.00th=[12608], '
'80.00th=[12736], '
'90.00th=[12736], '
'95.00th=[13376],\n'
' | 99.00th=[29824], '
'99.50th=[37120], '
'99.90th=[50944], '
'99.95th=[63744],\n'
' | 99.99th=[75264]\n'
' bw ( KiB/s): min=523344, '
'max=524897, per=100.00%, '
'avg=524383.50, stdev=486.06, '
'samples=14\n'
' iops : min=130836, '
'max=131224, avg=131095.86, '
'stdev=121.49, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.04%, 20=96.51%, 50=3.33%, '
'100=0.11%\n'
' lat (usec) : 250=0.01%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=32.27%, '
'sys=67.69%, ctx=43, majf=0, '
'minf=0\n'
' IO depths : 1=50.0%, '
'2=50.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=917591,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=512MiB/s (537MB/s), '
'512MiB/s-512MiB/s '
'(537MB/s-537MB/s), io=3584MiB '
'(3758MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_183.output'),
'state': {'begin': 1666726773.0111377,
'cmd': 'cat '
'/tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7946062088012695,
'end': 1666726773.805744,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_183.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_183.state')},
'cmd_184': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_184.output'),
'state': {'begin': 1666726775.8299632,
'cmd': 'rm '
'/tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7891864776611328,
'end': 1666726776.6191497,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_184.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_184.state')},
'cmd_185': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_185.output'),
'state': {'begin': 1666726776.621369,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.522973537445068,
'end': 1666726788.1443424,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_185.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_185.state')},
'cmd_186': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733877,\n'
' "timestamp_ms" : '
'1666733877524,\n'
' "time" : "Tue Oct 25 23:37:57 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3756961792,\n'
' "io_kbytes" : 3668908,\n'
' "bw_bytes" : 536632165,\n'
' "bw" : 524054,\n'
' "iops" : 131013.569490,\n'
' "runtime" : 7001,\n'
' "total_ios" : 917226,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1543,\n'
' "max" : 16892,\n'
' "mean" : 1635.534015,\n'
' "stddev" : '
'122.054841,\n'
' "N" : 917227\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 711,\n'
' "max" : 1299107,\n'
' "mean" : '
'12389.945191,\n'
' "stddev" : '
'4132.412283,\n'
' "N" : 917227,\n'
' "percentile" : {\n'
' "1.000000" : 10560,\n'
' "5.000000" : 10688,\n'
' "10.000000" : '
'10688,\n'
' "20.000000" : '
'10688,\n'
' "30.000000" : '
'10816,\n'
' "40.000000" : '
'11072,\n'
' "50.000000" : '
'12352,\n'
' "60.000000" : '
'12480,\n'
' "70.000000" : '
'12480,\n'
' "80.000000" : '
'12608,\n'
' "90.000000" : '
'12864,\n'
' "95.000000" : '
'13376,\n'
' "99.000000" : '
'29568,\n'
' "99.500000" : '
'36608,\n'
' "99.900000" : '
'42240,\n'
' "99.950000" : '
'62208,\n'
' "99.990000" : 75264\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9368,\n'
' "max" : 1300770,\n'
' "mean" : '
'14025.479206,\n'
' "stddev" : '
'4142.473008,\n'
' "N" : 917227\n'
' },\n'
' "bw_min" : 523160,\n'
' "bw_max" : 525571,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'524175.642857,\n'
' "bw_dev" : 851.873917,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 130790,\n'
' "iops_max" : 131392,\n'
' "iops_mean" : '
'131043.857143,\n'
' "iops_stddev" : '
'212.874053,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 35.357143,\n'
' "sys_cpu" : 64.614286,\n'
' "ctx" : 40,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 49.978522,\n'
' "2" : 50.021478,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.056911,\n'
' "20" : 96.554502,\n'
' "50" : 3.304638,\n'
' "100" : 0.081332,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=87073: Tue '
'Oct 25 23:37:57 2022\n'
' read: IOPS=131k, BW=512MiB/s '
'(537MB/s)(3583MiB/7001msec)\n'
' slat (nsec): min=1543, '
'max=16892, avg=1635.53, '
'stdev=122.05\n'
' clat (nsec): min=711, '
'max=1299.1k, avg=12389.95, '
'stdev=4132.41\n'
' lat (usec): min=9, '
'max=1300, avg=14.03, stdev= '
'4.14\n'
' clat percentiles (nsec):\n'
' | 1.00th=[10560], '
'5.00th=[10688], 10.00th=[10688], '
'20.00th=[10688],\n'
' | 30.00th=[10816], '
'40.00th=[11072], '
'50.00th=[12352], '
'60.00th=[12480],\n'
' | 70.00th=[12480], '
'80.00th=[12608], '
'90.00th=[12864], '
'95.00th=[13376],\n'
' | 99.00th=[29568], '
'99.50th=[36608], '
'99.90th=[42240], '
'99.95th=[62208],\n'
' | 99.99th=[75264]\n'
' bw ( KiB/s): min=523160, '
'max=525571, per=100.00%, '
'avg=524175.64, stdev=851.87, '
'samples=14\n'
' iops : min=130790, '
'max=131392, avg=131043.86, '
'stdev=212.87, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.06%, 20=96.55%, 50=3.30%, '
'100=0.08%\n'
' lat (usec) : 250=0.01%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=35.36%, '
'sys=64.61%, ctx=40, majf=0, '
'minf=0\n'
' IO depths : 1=50.0%, '
'2=50.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=917226,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=512MiB/s (537MB/s), '
'512MiB/s-512MiB/s '
'(537MB/s-537MB/s), io=3583MiB '
'(3757MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_186.output'),
'state': {'begin': 1666726788.146725,
'cmd': 'cat '
'/tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7905588150024414,
'end': 1666726788.9372838,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_186.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_186.state')},
'cmd_187': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_187.output'),
'state': {'begin': 1666726790.9592083,
'cmd': 'rm '
'/tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7852804660797119,
'end': 1666726791.7444887,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_187.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_187.state')},
'cmd_188': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_188.output'),
'state': {'begin': 1666726791.7467859,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.554159879684448,
'end': 1666726803.3009458,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_188.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_188.state')},
'cmd_189': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733892,\n'
' "timestamp_ms" : '
'1666733892681,\n'
' "time" : "Tue Oct 25 23:38:12 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'3749478400,\n'
' "io_kbytes" : 3661600,\n'
' "bw_bytes" : 535563262,\n'
' "bw" : 523010,\n'
' "iops" : 130752.606770,\n'
' "runtime" : 7001,\n'
' "total_ios" : 915399,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1552,\n'
' "max" : 175259,\n'
' "mean" : 1626.508294,\n'
' "stddev" : '
'223.366531,\n'
' "N" : 915399\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 711,\n'
' "max" : 181591,\n'
' "mean" : '
'12431.741996,\n'
' "stddev" : '
'3780.590270,\n'
' "N" : 915399,\n'
' "percentile" : {\n'
' "1.000000" : 10560,\n'
' "5.000000" : 10560,\n'
' "10.000000" : '
'10688,\n'
' "20.000000" : '
'10816,\n'
' "30.000000" : '
'10944,\n'
' "40.000000" : '
'11072,\n'
' "50.000000" : '
'12352,\n'
' "60.000000" : '
'12480,\n'
' "70.000000" : '
'12608,\n'
' "80.000000" : '
'12736,\n'
' "90.000000" : '
'12864,\n'
' "95.000000" : '
'13376,\n'
' "99.000000" : '
'29824,\n'
' "99.500000" : '
'37120,\n'
' "99.900000" : '
'50432,\n'
' "99.950000" : '
'63744,\n'
' "99.990000" : 80384\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9377,\n'
' "max" : 188053,\n'
' "mean" : '
'14058.250290,\n'
' "stddev" : '
'3793.051475,\n'
' "N" : 915399\n'
' },\n'
' "bw_min" : 521616,\n'
' "bw_max" : 524296,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'523131.285714,\n'
' "bw_dev" : 699.172965,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 130404,\n'
' "iops_max" : 131074,\n'
' "iops_mean" : '
'130782.785714,\n'
' "iops_stddev" : '
'174.475116,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 34.957143,\n'
' "sys_cpu" : 65.014286,\n'
' "ctx" : 17,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 49.974055,\n'
' "2" : 50.025945,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.063470,\n'
' "20" : 96.510811,\n'
' "50" : 3.320410,\n'
' "100" : 0.101267,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=87122: Tue '
'Oct 25 23:38:12 2022\n'
' read: IOPS=131k, BW=511MiB/s '
'(536MB/s)(3576MiB/7001msec)\n'
' slat (nsec): min=1552, '
'max=175259, avg=1626.51, '
'stdev=223.37\n'
' clat (nsec): min=711, '
'max=181591, avg=12431.74, '
'stdev=3780.59\n'
' lat (usec): min=9, max=188, '
'avg=14.06, stdev= 3.79\n'
' clat percentiles (nsec):\n'
' | 1.00th=[10560], '
'5.00th=[10560], 10.00th=[10688], '
'20.00th=[10816],\n'
' | 30.00th=[10944], '
'40.00th=[11072], '
'50.00th=[12352], '
'60.00th=[12480],\n'
' | 70.00th=[12608], '
'80.00th=[12736], '
'90.00th=[12864], '
'95.00th=[13376],\n'
' | 99.00th=[29824], '
'99.50th=[37120], '
'99.90th=[50432], '
'99.95th=[63744],\n'
' | 99.99th=[80384]\n'
' bw ( KiB/s): min=521616, '
'max=524296, per=100.00%, '
'avg=523131.29, stdev=699.17, '
'samples=14\n'
' iops : min=130404, '
'max=131074, avg=130782.79, '
'stdev=174.48, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.06%, 20=96.51%, 50=3.32%, '
'100=0.10%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=34.96%, '
'sys=65.01%, ctx=17, majf=0, '
'minf=0\n'
' IO depths : 1=50.0%, '
'2=50.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=915399,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=511MiB/s (536MB/s), '
'511MiB/s-511MiB/s '
'(536MB/s-536MB/s), io=3576MiB '
'(3749MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_189.output'),
'state': {'begin': 1666726803.303523,
'cmd': 'cat '
'/tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7989928722381592,
'end': 1666726804.102516,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_189.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_189.state')},
'cmd_19': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_19.output'),
'state': {'begin': 1666725984.2602887,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7942605018615723,
'end': 1666725985.0545492,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_19.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_19.state')},
'cmd_190': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_libaio-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_190.output'),
'state': {'begin': 1666726805.0965261,
'cmd': 'rm '
'/tmp/fio-output_4K_2_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7852075099945068,
'end': 1666726805.8817337,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_190.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_190.state')},
'cmd_191': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_191.output'),
'state': {'begin': 1666726805.888838,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_libaio-xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.097291946411133,
'end': 1666726816.98613,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_191.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_191.state')},
'cmd_192': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733906,\n'
' "timestamp_ms" : '
'1666733906474,\n'
' "time" : "Tue Oct 25 23:38:26 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2690433024,\n'
' "io_kbytes" : 2627376,\n'
' "bw_bytes" : 384292675,\n'
' "bw" : 375285,\n'
' "iops" : 93821.311241,\n'
' "runtime" : 7001,\n'
' "total_ios" : 656843,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2504,\n'
' "max" : 178896,\n'
' "mean" : 2877.458288,\n'
' "stddev" : '
'468.361840,\n'
' "N" : 656843\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 951,\n'
' "max" : 1352827,\n'
' "mean" : '
'17919.024456,\n'
' "stddev" : '
'5851.439789,\n'
' "N" : 656844,\n'
' "percentile" : {\n'
' "1.000000" : 12736,\n'
' "5.000000" : 12864,\n'
' "10.000000" : '
'12992,\n'
' "20.000000" : '
'12992,\n'
' "30.000000" : '
'13120,\n'
' "40.000000" : '
'13120,\n'
' "50.000000" : '
'21376,\n'
' "60.000000" : '
'21376,\n'
' "70.000000" : '
'21376,\n'
' "80.000000" : '
'21632,\n'
' "90.000000" : '
'21632,\n'
' "95.000000" : '
'22144,\n'
' "99.000000" : '
'35072,\n'
' "99.500000" : '
'40192,\n'
' "99.900000" : '
'50944,\n'
' "99.950000" : '
'64256,\n'
' "99.990000" : 77312\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12153,\n'
' "max" : 1355933,\n'
' "mean" : '
'20796.483029,\n'
' "stddev" : '
'6068.884642,\n'
' "N" : 656844\n'
' },\n'
' "bw_min" : 374056,\n'
' "bw_max" : 377312,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'375377.071429,\n'
' "bw_dev" : 779.433371,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 93514,\n'
' "iops_max" : 94328,\n'
' "iops_mean" : '
'93844.214286,\n'
' "iops_stddev" : '
'194.825436,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.200000,\n'
' "sys_cpu" : 45.857143,\n'
' "ctx" : 328203,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 47.701962,\n'
' "50" : 52.184160,\n'
' "100" : 0.103678,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=87161: Tue '
'Oct 25 23:38:26 2022\n'
' read: IOPS=93.8k, BW=366MiB/s '
'(384MB/s)(2566MiB/7001msec)\n'
' slat (nsec): min=2504, '
'max=178896, avg=2877.46, '
'stdev=468.36\n'
' clat (nsec): min=951, '
'max=1352.8k, avg=17919.02, '
'stdev=5851.44\n'
' lat (usec): min=12, '
'max=1355, avg=20.80, stdev= '
'6.07\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12736], '
'5.00th=[12864], 10.00th=[12992], '
'20.00th=[12992],\n'
' | 30.00th=[13120], '
'40.00th=[13120], '
'50.00th=[21376], '
'60.00th=[21376],\n'
' | 70.00th=[21376], '
'80.00th=[21632], '
'90.00th=[21632], '
'95.00th=[22144],\n'
' | 99.00th=[35072], '
'99.50th=[40192], '
'99.90th=[50944], '
'99.95th=[64256],\n'
' | 99.99th=[77312]\n'
' bw ( KiB/s): min=374056, '
'max=377312, per=100.00%, '
'avg=375377.07, stdev=779.43, '
'samples=14\n'
' iops : min=93514, '
'max=94328, avg=93844.21, '
'stdev=194.83, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'10=0.01%, 20=47.70%, 50=52.18%, '
'100=0.10%\n'
' lat (usec) : 250=0.01%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=15.20%, '
'sys=45.86%, ctx=328203, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=656843,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=366MiB/s (384MB/s), '
'366MiB/s-366MiB/s '
'(384MB/s-384MB/s), io=2566MiB '
'(2690MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_192.output'),
'state': {'begin': 1666726816.9890933,
'cmd': 'cat '
'/tmp/fio-output_4K_2_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7825303077697754,
'end': 1666726817.7716236,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_192.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_192.state')},
'cmd_193': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_libaio-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_193.output'),
'state': {'begin': 1666726818.7582977,
'cmd': 'rm '
'/tmp/fio-output_4K_2_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.78106689453125,
'end': 1666726819.5393646,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_193.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_193.state')},
'cmd_194': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_194.output'),
'state': {'begin': 1666726819.541263,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_libaio-xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.086409330368042,
'end': 1666726830.6276724,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_194.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_194.state')},
'cmd_195': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733920,\n'
' "timestamp_ms" : '
'1666733920115,\n'
' "time" : "Tue Oct 25 23:38:40 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2698158080,\n'
' "io_kbytes" : 2634920,\n'
' "bw_bytes" : 385396097,\n'
' "bw" : 376363,\n'
' "iops" : 94090.701328,\n'
' "runtime" : 7001,\n'
' "total_ios" : 658729,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2534,\n'
' "max" : 47179,\n'
' "mean" : 2900.782150,\n'
' "stddev" : '
'418.117679,\n'
' "N" : 658729\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 921,\n'
' "max" : 118352,\n'
' "mean" : '
'17829.021652,\n'
' "stddev" : '
'5403.400026,\n'
' "N" : 658730,\n'
' "percentile" : {\n'
' "1.000000" : 12608,\n'
' "5.000000" : 12736,\n'
' "10.000000" : '
'12736,\n'
' "20.000000" : '
'12864,\n'
' "30.000000" : '
'12864,\n'
' "40.000000" : '
'12992,\n'
' "50.000000" : '
'21376,\n'
' "60.000000" : '
'21376,\n'
' "70.000000" : '
'21376,\n'
' "80.000000" : '
'21632,\n'
' "90.000000" : '
'21632,\n'
' "95.000000" : '
'22144,\n'
' "99.000000" : '
'34560,\n'
' "99.500000" : '
'39680,\n'
' "99.900000" : '
'48896,\n'
' "99.950000" : '
'61184,\n'
' "99.990000" : 76288\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12253,\n'
' "max" : 121407,\n'
' "mean" : '
'20729.804006,\n'
' "stddev" : '
'5628.504721,\n'
' "N" : 658730\n'
' },\n'
' "bw_min" : 375336,\n'
' "bw_max" : 377432,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'376463.000000,\n'
' "bw_dev" : 622.233570,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 93834,\n'
' "iops_max" : 94358,\n'
' "iops_mean" : '
'94115.714286,\n'
' "iops_stddev" : '
'155.503860,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.242857,\n'
' "sys_cpu" : 46.571429,\n'
' "ctx" : 329136,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010019,\n'
' "20" : 47.720686,\n'
' "50" : 52.176995,\n'
' "100" : 0.090629,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=87201: Tue '
'Oct 25 23:38:40 2022\n'
' read: IOPS=94.1k, BW=368MiB/s '
'(385MB/s)(2573MiB/7001msec)\n'
' slat (nsec): min=2534, '
'max=47179, avg=2900.78, '
'stdev=418.12\n'
' clat (nsec): min=921, '
'max=118352, avg=17829.02, '
'stdev=5403.40\n'
' lat (usec): min=12, '
'max=121, avg=20.73, stdev= 5.63\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12608], '
'5.00th=[12736], 10.00th=[12736], '
'20.00th=[12864],\n'
' | 30.00th=[12864], '
'40.00th=[12992], '
'50.00th=[21376], '
'60.00th=[21376],\n'
' | 70.00th=[21376], '
'80.00th=[21632], '
'90.00th=[21632], '
'95.00th=[22144],\n'
' | 99.00th=[34560], '
'99.50th=[39680], '
'99.90th=[48896], '
'99.95th=[61184],\n'
' | 99.99th=[76288]\n'
' bw ( KiB/s): min=375336, '
'max=377432, per=100.00%, '
'avg=376463.00, stdev=622.23, '
'samples=14\n'
' iops : min=93834, '
'max=94358, avg=94115.71, '
'stdev=155.50, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.01%, 20=47.72%, '
'50=52.18%\n'
' lat (usec) : 100=0.09%, '
'250=0.01%\n'
' cpu : usr=15.24%, '
'sys=46.57%, ctx=329136, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=658729,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=368MiB/s (385MB/s), '
'368MiB/s-368MiB/s '
'(385MB/s-385MB/s), io=2573MiB '
'(2698MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_195.output'),
'state': {'begin': 1666726830.6348295,
'cmd': 'cat '
'/tmp/fio-output_4K_2_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7972743511199951,
'end': 1666726831.4321039,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_195.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_195.state')},
'cmd_196': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_2_libaio-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_196.output'),
'state': {'begin': 1666726832.428195,
'cmd': 'rm '
'/tmp/fio-output_4K_2_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7885875701904297,
'end': 1666726833.2167826,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_196.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_196.state')},
'cmd_197': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_197.output'),
'state': {'begin': 1666726833.2185743,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_2_libaio-xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.09030032157898,
'end': 1666726844.3088746,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_197.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_197.state')},
'cmd_198': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733933,\n'
' "timestamp_ms" : '
'1666733933796,\n'
' "time" : "Tue Oct 25 23:38:53 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2710253568,\n'
' "io_kbytes" : 2646732,\n'
' "bw_bytes" : 387123777,\n'
' "bw" : 378050,\n'
' "iops" : 94512.498215,\n'
' "runtime" : 7001,\n'
' "total_ios" : 661682,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2494,\n'
' "max" : 179777,\n'
' "mean" : 2858.011643,\n'
' "stddev" : '
'548.309512,\n'
' "N" : 661682\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 902,\n'
' "max" : 195327,\n'
' "mean" : '
'17806.183939,\n'
' "stddev" : '
'5345.044073,\n'
' "N" : 661683,\n'
' "percentile" : {\n'
' "1.000000" : 12736,\n'
' "5.000000" : 12864,\n'
' "10.000000" : '
'12864,\n'
' "20.000000" : '
'12992,\n'
' "30.000000" : '
'12992,\n'
' "40.000000" : '
'13120,\n'
' "50.000000" : '
'21120,\n'
' "60.000000" : '
'21376,\n'
' "70.000000" : '
'21376,\n'
' "80.000000" : '
'21376,\n'
' "90.000000" : '
'21376,\n'
' "95.000000" : '
'21888,\n'
' "99.000000" : '
'35072,\n'
' "99.500000" : '
'40192,\n'
' "99.900000" : '
'50944,\n'
' "99.950000" : '
'64768,\n'
' "99.990000" : 77312\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12413,\n'
' "max" : 201298,\n'
' "mean" : '
'20664.195956,\n'
' "stddev" : '
'5573.558208,\n'
' "N" : 661683\n'
' },\n'
' "bw_min" : 377240,\n'
' "bw_max" : 379552,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'378145.928571,\n'
' "bw_dev" : 652.061638,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 94310,\n'
' "iops_max" : 94888,\n'
' "iops_mean" : '
'94536.428571,\n'
' "iops_stddev" : '
'163.055778,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.442857,\n'
' "sys_cpu" : 43.571429,\n'
' "ctx" : 330603,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 47.726551,\n'
' "50" : 52.157683,\n'
' "100" : 0.103373,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=87241: Tue '
'Oct 25 23:38:53 2022\n'
' read: IOPS=94.5k, BW=369MiB/s '
'(387MB/s)(2585MiB/7001msec)\n'
' slat (nsec): min=2494, '
'max=179777, avg=2858.01, '
'stdev=548.31\n'
' clat (nsec): min=902, '
'max=195327, avg=17806.18, '
'stdev=5345.04\n'
' lat (usec): min=12, '
'max=201, avg=20.66, stdev= 5.57\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12736], '
'5.00th=[12864], 10.00th=[12864], '
'20.00th=[12992],\n'
' | 30.00th=[12992], '
'40.00th=[13120], '
'50.00th=[21120], '
'60.00th=[21376],\n'
' | 70.00th=[21376], '
'80.00th=[21376], '
'90.00th=[21376], '
'95.00th=[21888],\n'
' | 99.00th=[35072], '
'99.50th=[40192], '
'99.90th=[50944], '
'99.95th=[64768],\n'
' | 99.99th=[77312]\n'
' bw ( KiB/s): min=377240, '
'max=379552, per=100.00%, '
'avg=378145.93, stdev=652.06, '
'samples=14\n'
' iops : min=94310, '
'max=94888, avg=94536.43, '
'stdev=163.06, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'10=0.01%, 20=47.73%, 50=52.16%, '
'100=0.10%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=17.44%, '
'sys=43.57%, ctx=330603, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=661682,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=369MiB/s (387MB/s), '
'369MiB/s-369MiB/s '
'(387MB/s-387MB/s), io=2585MiB '
'(2710MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_198.output'),
'state': {'begin': 1666726844.3117392,
'cmd': 'cat '
'/tmp/fio-output_4K_2_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.794931173324585,
'end': 1666726845.1066704,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_198.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_198.state')},
'cmd_199': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_199.output'),
'state': {'begin': 1666726846.1032386,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7885692119598389,
'end': 1666726846.8918078,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_199.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_199.state')},
'cmd_20': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_20.output'),
'state': {'begin': 1666725985.05676,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.475365400314331,
'end': 1666725996.5321255,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_20.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_20.state')},
'cmd_200': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_200.output'),
'state': {'begin': 1666726846.894031,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt"',
'cwd': 'None',
'elapsed': 11.049890756607056,
'end': 1666726857.9439218,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_200.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_200.state')},
'cmd_201': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, '
'ioengine=io_uring_cmd, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733947,\n'
' "timestamp_ms" : '
'1666733947431,\n'
' "time" : "Tue Oct 25 23:39:07 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'4239355904,\n'
' "io_kbytes" : 4139996,\n'
' "bw_bytes" : 605535766,\n'
' "bw" : 591343,\n'
' "iops" : 147835.309242,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1034995,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1863,\n'
' "max" : 68268,\n'
' "mean" : 2134.228236,\n'
' "stddev" : '
'423.418030,\n'
' "N" : 1034996\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 210,\n'
' "max" : 170530,\n'
' "mean" : '
'24422.255110,\n'
' "stddev" : '
'7557.409280,\n'
' "N" : 1034998,\n'
' "percentile" : {\n'
' "1.000000" : 14528,\n'
' "5.000000" : 17024,\n'
' "10.000000" : '
'17024,\n'
' "20.000000" : '
'17792,\n'
' "30.000000" : '
'18048,\n'
' "40.000000" : '
'18560,\n'
' "50.000000" : '
'28032,\n'
' "60.000000" : '
'28032,\n'
' "70.000000" : '
'28288,\n'
' "80.000000" : '
'28800,\n'
' "90.000000" : '
'29056,\n'
' "95.000000" : '
'34048,\n'
' "99.000000" : '
'51968,\n'
' "99.500000" : '
'53504,\n'
' "99.900000" : '
'82432,\n'
' "99.950000" : '
'97792,\n'
' "99.990000" : '
'112128\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 13415,\n'
' "max" : 172494,\n'
' "mean" : '
'26556.483509,\n'
' "stddev" : '
'7660.333802,\n'
' "N" : 1034998\n'
' },\n'
' "bw_min" : 590184,\n'
' "bw_max" : 592416,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'591485.071429,\n'
' "bw_dev" : 642.692948,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 147546,\n'
' "iops_max" : 148104,\n'
' "iops_mean" : '
'147871.214286,\n'
' "iops_stddev" : '
'160.657389,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.742857,\n'
' "sys_cpu" : 57.785714,\n'
' "ctx" : 266084,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 44.522920,\n'
' "50" : 53.856106,\n'
' "100" : 1.573244,\n'
' "250" : 0.045604,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=87281: Tue Oct 25 23:39:07 '
'2022\n'
' read: IOPS=148k, BW=577MiB/s '
'(606MB/s)(4043MiB/7001msec)\n'
' slat (nsec): min=1863, '
'max=68268, avg=2134.23, '
'stdev=423.42\n'
' clat (nsec): min=210, '
'max=170530, avg=24422.26, '
'stdev=7557.41\n'
' lat (usec): min=13, '
'max=172, avg=26.56, stdev= 7.66\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 15], '
'5.00th=[ 18], 10.00th=[ 18], '
'20.00th=[ 18],\n'
' | 30.00th=[ 19], '
'40.00th=[ 19], 50.00th=[ '
'29], 60.00th=[ 29],\n'
' | 70.00th=[ 29], '
'80.00th=[ 29], 90.00th=[ '
'30], 95.00th=[ 35],\n'
' | 99.00th=[ 52], '
'99.50th=[ 54], 99.90th=[ '
'83], 99.95th=[ 98],\n'
' | 99.99th=[ 113]\n'
' bw ( KiB/s): min=590184, '
'max=592416, per=100.00%, '
'avg=591485.07, stdev=642.69, '
'samples=14\n'
' iops : min=147546, '
'max=148104, avg=147871.21, '
'stdev=160.66, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=44.52%, 50=53.86%, '
'100=1.57%\n'
' lat (usec) : 250=0.05%\n'
' cpu : usr=14.74%, '
'sys=57.79%, ctx=266084, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1034995,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=577MiB/s (606MB/s), '
'577MiB/s-577MiB/s '
'(606MB/s-606MB/s), io=4043MiB '
'(4239MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_201.output'),
'state': {'begin': 1666726857.9465978,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7997150421142578,
'end': 1666726858.7463129,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_201.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_201.state')},
'cmd_202': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_202.output'),
'state': {'begin': 1666726859.737454,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.794926643371582,
'end': 1666726860.5323806,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_202.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_202.state')},
'cmd_203': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_203.output'),
'state': {'begin': 1666726860.5341961,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt"',
'cwd': 'None',
'elapsed': 11.052112102508545,
'end': 1666726871.5863082,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_203.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_203.state')},
'cmd_204': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, '
'ioengine=io_uring_cmd, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733961,\n'
' "timestamp_ms" : '
'1666733961073,\n'
' "time" : "Tue Oct 25 23:39:21 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'4248465408,\n'
' "io_kbytes" : 4148892,\n'
' "bw_bytes" : 606836938,\n'
' "bw" : 592614,\n'
' "iops" : 148152.978146,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1037219,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1874,\n'
' "max" : 27602,\n'
' "mean" : 2143.573774,\n'
' "stddev" : '
'413.583393,\n'
' "N" : 1037220\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2985,\n'
' "max" : 204163,\n'
' "mean" : '
'24353.154686,\n'
' "stddev" : '
'7561.186398,\n'
' "N" : 1037222,\n'
' "percentile" : {\n'
' "1.000000" : 16768,\n'
' "5.000000" : 17024,\n'
' "10.000000" : '
'17024,\n'
' "20.000000" : '
'17792,\n'
' "30.000000" : '
'17792,\n'
' "40.000000" : '
'18560,\n'
' "50.000000" : '
'28032,\n'
' "60.000000" : '
'28032,\n'
' "70.000000" : '
'28288,\n'
' "80.000000" : '
'28800,\n'
' "90.000000" : '
'29056,\n'
' "95.000000" : '
'34048,\n'
' "99.000000" : '
'51968,\n'
' "99.500000" : '
'53504,\n'
' "99.900000" : '
'81408,\n'
' "99.950000" : '
'96768,\n'
' "99.990000" : '
'113152\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 13285,\n'
' "max" : 206557,\n'
' "mean" : '
'26496.728306,\n'
' "stddev" : '
'7666.456687,\n'
' "N" : 1037222\n'
' },\n'
' "bw_min" : 591632,\n'
' "bw_max" : 593816,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'592752.785714,\n'
' "bw_dev" : 705.015022,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 147908,\n'
' "iops_max" : 148454,\n'
' "iops_mean" : '
'148188.142857,\n'
' "iops_stddev" : '
'176.173803,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 16.328571,\n'
' "sys_cpu" : 55.942857,\n'
' "ctx" : 266741,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 45.803249,\n'
' "50" : 52.586484,\n'
' "100" : 1.563893,\n'
' "250" : 0.044157,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=87319: Tue Oct 25 23:39:21 '
'2022\n'
' read: IOPS=148k, BW=579MiB/s '
'(607MB/s)(4052MiB/7001msec)\n'
' slat (nsec): min=1874, '
'max=27602, avg=2143.57, '
'stdev=413.58\n'
' clat (usec): min=2, max=204, '
'avg=24.35, stdev= 7.56\n'
' lat (usec): min=13, '
'max=206, avg=26.50, stdev= 7.67\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 17], '
'5.00th=[ 18], 10.00th=[ 18], '
'20.00th=[ 18],\n'
' | 30.00th=[ 18], '
'40.00th=[ 19], 50.00th=[ '
'29], 60.00th=[ 29],\n'
' | 70.00th=[ 29], '
'80.00th=[ 29], 90.00th=[ '
'30], 95.00th=[ 35],\n'
' | 99.00th=[ 52], '
'99.50th=[ 54], 99.90th=[ '
'82], 99.95th=[ 97],\n'
' | 99.99th=[ 114]\n'
' bw ( KiB/s): min=591632, '
'max=593816, per=100.00%, '
'avg=592752.79, stdev=705.02, '
'samples=14\n'
' iops : min=147908, '
'max=148454, avg=148188.14, '
'stdev=176.17, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=45.80%, 50=52.59%, '
'100=1.56%\n'
' lat (usec) : 250=0.04%\n'
' cpu : usr=16.33%, '
'sys=55.94%, ctx=266741, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1037219,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=579MiB/s (607MB/s), '
'579MiB/s-579MiB/s '
'(607MB/s-607MB/s), io=4052MiB '
'(4248MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_204.output'),
'state': {'begin': 1666726871.5889437,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.793954610824585,
'end': 1666726872.3828983,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_204.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_204.state')},
'cmd_205': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_205.output'),
'state': {'begin': 1666726873.4409869,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7898757457733154,
'end': 1666726874.2308626,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_205.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_205.state')},
'cmd_206': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_206.output'),
'state': {'begin': 1666726874.2328246,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt"',
'cwd': 'None',
'elapsed': 11.050575256347656,
'end': 1666726885.2833998,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_206.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_206.state')},
'cmd_207': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, '
'ioengine=io_uring_cmd, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733974,\n'
' "timestamp_ms" : '
'1666733974770,\n'
' "time" : "Tue Oct 25 23:39:34 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'4263362560,\n'
' "io_kbytes" : 4163440,\n'
' "bw_bytes" : 608964799,\n'
' "bw" : 594692,\n'
' "iops" : 148672.618197,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1040857,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1883,\n'
' "max" : 30347,\n'
' "mean" : 2119.325331,\n'
' "stddev" : '
'420.060474,\n'
' "N" : 1040857\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 251,\n'
' "max" : 149521,\n'
' "mean" : '
'24289.765743,\n'
' "stddev" : '
'7494.997171,\n'
' "N" : 1040860,\n'
' "percentile" : {\n'
' "1.000000" : 15168,\n'
' "5.000000" : 17024,\n'
' "10.000000" : '
'17024,\n'
' "20.000000" : '
'17792,\n'
' "30.000000" : '
'17792,\n'
' "40.000000" : '
'18560,\n'
' "50.000000" : '
'27776,\n'
' "60.000000" : '
'28032,\n'
' "70.000000" : '
'28288,\n'
' "80.000000" : '
'28800,\n'
' "90.000000" : '
'29056,\n'
' "95.000000" : '
'34048,\n'
' "99.000000" : '
'51968,\n'
' "99.500000" : '
'53504,\n'
' "99.900000" : '
'81408,\n'
' "99.950000" : '
'97792,\n'
' "99.990000" : '
'113152\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 14096,\n'
' "max" : 151475,\n'
' "mean" : '
'26409.091097,\n'
' "stddev" : '
'7588.653299,\n'
' "N" : 1040860\n'
' },\n'
' "bw_min" : 593880,\n'
' "bw_max" : 595576,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'594850.642857,\n'
' "bw_dev" : 491.431129,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 148470,\n'
' "iops_max" : 148894,\n'
' "iops_mean" : '
'148712.642857,\n'
' "iops_stddev" : '
'122.870543,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.671429,\n'
' "sys_cpu" : 58.242857,\n'
' "ctx" : 267603,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 45.125891,\n'
' "50" : 53.340180,\n'
' "100" : 1.487716,\n'
' "250" : 0.043714,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=87357: Tue Oct 25 23:39:34 '
'2022\n'
' read: IOPS=149k, BW=581MiB/s '
'(609MB/s)(4066MiB/7001msec)\n'
' slat (nsec): min=1883, '
'max=30347, avg=2119.33, '
'stdev=420.06\n'
' clat (nsec): min=251, '
'max=149521, avg=24289.77, '
'stdev=7495.00\n'
' lat (usec): min=14, '
'max=151, avg=26.41, stdev= 7.59\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 16], '
'5.00th=[ 18], 10.00th=[ 18], '
'20.00th=[ 18],\n'
' | 30.00th=[ 18], '
'40.00th=[ 19], 50.00th=[ '
'28], 60.00th=[ 29],\n'
' | 70.00th=[ 29], '
'80.00th=[ 29], 90.00th=[ '
'30], 95.00th=[ 35],\n'
' | 99.00th=[ 52], '
'99.50th=[ 54], 99.90th=[ '
'82], 99.95th=[ 98],\n'
' | 99.99th=[ 114]\n'
' bw ( KiB/s): min=593880, '
'max=595576, per=100.00%, '
'avg=594850.64, stdev=491.43, '
'samples=14\n'
' iops : min=148470, '
'max=148894, avg=148712.64, '
'stdev=122.87, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=45.13%, 50=53.34%, '
'100=1.49%\n'
' lat (usec) : 250=0.04%\n'
' cpu : usr=13.67%, '
'sys=58.24%, ctx=267603, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1040857,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=581MiB/s (609MB/s), '
'581MiB/s-581MiB/s '
'(609MB/s-609MB/s), io=4066MiB '
'(4263MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_207.output'),
'state': {'begin': 1666726885.2859445,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7888407707214355,
'end': 1666726886.0747852,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_207.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_207.state')},
'cmd_208': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_208.output'),
'state': {'begin': 1666726887.0683186,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7800571918487549,
'end': 1666726887.8483758,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_208.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_208.state')},
'cmd_209': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_209.output'),
'state': {'begin': 1666726887.8502033,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.055967092514038,
'end': 1666726898.9061704,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_209.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_209.state')},
'cmd_21': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733085,\n'
' "timestamp_ms" : '
'1666733085963,\n'
' "time" : "Tue Oct 25 23:24:45 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2405982208,\n'
' "io_kbytes" : 2349592,\n'
' "bw_bytes" : 343662649,\n'
' "bw" : 335608,\n'
' "iops" : 83902.013998,\n'
' "runtime" : 7001,\n'
' "total_ios" : 587398,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 711,\n'
' "max" : 4368,\n'
' "mean" : 924.473922,\n'
' "stddev" : 69.051652,\n'
' "N" : 587398\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 9077,\n'
' "max" : 113724,\n'
' "mean" : 10706.197505,\n'
' "stddev" : '
'2217.342043,\n'
' "N" : 587398,\n'
' "percentile" : {\n'
' "1.000000" : 9920,\n'
' "5.000000" : 10048,\n'
' "10.000000" : 10048,\n'
' "20.000000" : 10176,\n'
' "30.000000" : 10304,\n'
' "40.000000" : 10304,\n'
' "50.000000" : 10432,\n'
' "60.000000" : 10432,\n'
' "70.000000" : 10560,\n'
' "80.000000" : 10560,\n'
' "90.000000" : 10688,\n'
' "95.000000" : 11072,\n'
' "99.000000" : 20096,\n'
' "99.500000" : 27520,\n'
' "99.900000" : 30080,\n'
' "99.950000" : 48384,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10510,\n'
' "max" : 114625,\n'
' "mean" : 11630.671427,\n'
' "stddev" : '
'2218.545088,\n'
' "N" : 587398\n'
' },\n'
' "bw_min" : 335280,\n'
' "bw_max" : 336168,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'335695.428571,\n'
' "bw_dev" : 291.592369,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 83820,\n'
' "iops_max" : 84042,\n'
' "iops_mean" : '
'83923.857143,\n'
' "iops_stddev" : '
'72.898092,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 16,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 3.081897,\n'
' "20" : 95.920653,\n'
' "50" : 0.950122,\n'
' "100" : 0.047157,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=84633: Tue Oct 25 23:24:45 '
'2022\n'
' read: IOPS=83.9k, BW=328MiB/s '
'(344MB/s)(2295MiB/7001msec)\n'
' slat (nsec): min=711, '
'max=4368, avg=924.47, '
'stdev=69.05\n'
' clat (usec): min=9, max=113, '
'avg=10.71, stdev= 2.22\n'
' lat (usec): min=10, max=114, '
'avg=11.63, stdev= 2.22\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9920], '
'5.00th=[10048], 10.00th=[10048], '
'20.00th=[10176],\n'
' | 30.00th=[10304], '
'40.00th=[10304], 50.00th=[10432], '
'60.00th=[10432],\n'
' | 70.00th=[10560], '
'80.00th=[10560], 90.00th=[10688], '
'95.00th=[11072],\n'
' | 99.00th=[20096], '
'99.50th=[27520], 99.90th=[30080], '
'99.95th=[48384],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=335280, '
'max=336168, per=100.00%, '
'avg=335695.43, stdev=291.59, '
'samples=14\n'
' iops : min=83820, '
'max=84042, avg=83923.86, '
'stdev=72.90, samples=14\n'
' lat (usec) : 10=3.08%, '
'20=95.92%, 50=0.95%, 100=0.05%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=16, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=587398,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=328MiB/s (344MB/s), '
'328MiB/s-328MiB/s '
'(344MB/s-344MB/s), io=2295MiB '
'(2406MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_21.output'),
'state': {'begin': 1666725996.534893,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7965598106384277,
'end': 1666725997.3314528,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_21.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_21.state')},
'cmd_210': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733988,\n'
' "timestamp_ms" : '
'1666733988393,\n'
' "time" : "Tue Oct 25 23:39:48 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'4244062208,\n'
' "io_kbytes" : 4144592,\n'
' "bw_bytes" : 606208000,\n'
' "bw" : 592000,\n'
' "iops" : 147999.428653,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1036144,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1973,\n'
' "max" : 32401,\n'
' "mean" : 2217.550113,\n'
' "stddev" : '
'422.893349,\n'
' "N" : 1036145\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 260,\n'
' "max" : 149982,\n'
' "mean" : '
'24348.226217,\n'
' "stddev" : '
'7711.887494,\n'
' "N" : 1036147,\n'
' "percentile" : {\n'
' "1.000000" : 16512,\n'
' "5.000000" : 16768,\n'
' "10.000000" : '
'16768,\n'
' "20.000000" : '
'17536,\n'
' "30.000000" : '
'17536,\n'
' "40.000000" : '
'18304,\n'
' "50.000000" : '
'28288,\n'
' "60.000000" : '
'28288,\n'
' "70.000000" : '
'28544,\n'
' "80.000000" : '
'29056,\n'
' "90.000000" : '
'29312,\n'
' "95.000000" : '
'34048,\n'
' "99.000000" : '
'51968,\n'
' "99.500000" : '
'53504,\n'
' "99.900000" : '
'80384,\n'
' "99.950000" : '
'96768,\n'
' "99.990000" : '
'113152\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 13815,\n'
' "max" : 152456,\n'
' "mean" : '
'26565.776468,\n'
' "stddev" : '
'7822.726275,\n'
' "N" : 1036147\n'
' },\n'
' "bw_min" : 590720,\n'
' "bw_max" : 593480,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'592140.500000,\n'
' "bw_dev" : 670.367379,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 147680,\n'
' "iops_max" : 148370,\n'
' "iops_mean" : '
'148035.071429,\n'
' "iops_stddev" : '
'167.612403,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.842857,\n'
' "sys_cpu" : 56.142857,\n'
' "ctx" : 265858,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 45.727042,\n'
' "50" : 52.683121,\n'
' "100" : 1.546310,\n'
' "250" : 0.040824,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=87395: Tue '
'Oct 25 23:39:48 2022\n'
' read: IOPS=148k, BW=578MiB/s '
'(606MB/s)(4047MiB/7001msec)\n'
' slat (nsec): min=1973, '
'max=32401, avg=2217.55, '
'stdev=422.89\n'
' clat (nsec): min=260, '
'max=149982, avg=24348.23, '
'stdev=7711.89\n'
' lat (usec): min=13, '
'max=152, avg=26.57, stdev= 7.82\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 17], '
'5.00th=[ 17], 10.00th=[ 17], '
'20.00th=[ 18],\n'
' | 30.00th=[ 18], '
'40.00th=[ 19], 50.00th=[ '
'29], 60.00th=[ 29],\n'
' | 70.00th=[ 29], '
'80.00th=[ 30], 90.00th=[ '
'30], 95.00th=[ 35],\n'
' | 99.00th=[ 52], '
'99.50th=[ 54], 99.90th=[ '
'81], 99.95th=[ 97],\n'
' | 99.99th=[ 114]\n'
' bw ( KiB/s): min=590720, '
'max=593480, per=100.00%, '
'avg=592140.50, stdev=670.37, '
'samples=14\n'
' iops : min=147680, '
'max=148370, avg=148035.07, '
'stdev=167.61, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=45.73%, 50=52.68%, '
'100=1.55%\n'
' lat (usec) : 250=0.04%\n'
' cpu : usr=17.84%, '
'sys=56.14%, ctx=265858, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1036144,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=578MiB/s (606MB/s), '
'578MiB/s-578MiB/s '
'(606MB/s-606MB/s), io=4047MiB '
'(4244MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_210.output'),
'state': {'begin': 1666726898.9089272,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.9105038642883301,
'end': 1666726899.819431,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_210.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_210.state')},
'cmd_211': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_211.output'),
'state': {'begin': 1666726900.888153,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8740639686584473,
'end': 1666726901.762217,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_211.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_211.state')},
'cmd_212': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_212.output'),
'state': {'begin': 1666726901.7642074,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.154089212417603,
'end': 1666726912.9182966,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_212.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_212.state')},
'cmd_213': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734002,\n'
' "timestamp_ms" : '
'1666734002404,\n'
' "time" : "Tue Oct 25 23:40:02 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'4233482240,\n'
' "io_kbytes" : 4134260,\n'
' "bw_bytes" : 604696791,\n'
' "bw" : 590524,\n'
' "iops" : 147630.624197,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1033562,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1973,\n'
' "max" : 28543,\n'
' "mean" : 2227.234882,\n'
' "stddev" : '
'433.965898,\n'
' "N" : 1033562\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 291,\n'
' "max" : 150883,\n'
' "mean" : '
'24397.982275,\n'
' "stddev" : '
'7660.972958,\n'
' "N" : 1033565,\n'
' "percentile" : {\n'
' "1.000000" : 16512,\n'
' "5.000000" : 16768,\n'
' "10.000000" : '
'16768,\n'
' "20.000000" : '
'17536,\n'
' "30.000000" : '
'17792,\n'
' "40.000000" : '
'18304,\n'
' "50.000000" : '
'28288,\n'
' "60.000000" : '
'28544,\n'
' "70.000000" : '
'28544,\n'
' "80.000000" : '
'29056,\n'
' "90.000000" : '
'29312,\n'
' "95.000000" : '
'34048,\n'
' "99.000000" : '
'51968,\n'
' "99.500000" : '
'52992,\n'
' "99.900000" : '
'76288,\n'
' "99.950000" : '
'94720,\n'
' "99.990000" : '
'112128\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 13616,\n'
' "max" : 153037,\n'
' "mean" : '
'26625.217129,\n'
' "stddev" : '
'7783.374398,\n'
' "N" : 1033565\n'
' },\n'
' "bw_min" : 589488,\n'
' "bw_max" : 591486,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'590663.285714,\n'
' "bw_dev" : 586.088787,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 147372,\n'
' "iops_max" : 147871,\n'
' "iops_mean" : '
'147665.785714,\n'
' "iops_stddev" : '
'146.583753,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 18.671429,\n'
' "sys_cpu" : 55.157143,\n'
' "ctx" : 265038,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 45.679117,\n'
' "50" : 52.747102,\n'
' "100" : 1.533628,\n'
' "250" : 0.037540,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=87435: Tue '
'Oct 25 23:40:02 2022\n'
' read: IOPS=148k, BW=577MiB/s '
'(605MB/s)(4037MiB/7001msec)\n'
' slat (nsec): min=1973, '
'max=28543, avg=2227.23, '
'stdev=433.97\n'
' clat (nsec): min=291, '
'max=150883, avg=24397.98, '
'stdev=7660.97\n'
' lat (usec): min=13, '
'max=153, avg=26.63, stdev= 7.78\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 17], '
'5.00th=[ 17], 10.00th=[ 17], '
'20.00th=[ 18],\n'
' | 30.00th=[ 18], '
'40.00th=[ 19], 50.00th=[ '
'29], 60.00th=[ 29],\n'
' | 70.00th=[ 29], '
'80.00th=[ 30], 90.00th=[ '
'30], 95.00th=[ 35],\n'
' | 99.00th=[ 52], '
'99.50th=[ 53], 99.90th=[ '
'77], 99.95th=[ 95],\n'
' | 99.99th=[ 113]\n'
' bw ( KiB/s): min=589488, '
'max=591486, per=100.00%, '
'avg=590663.29, stdev=586.09, '
'samples=14\n'
' iops : min=147372, '
'max=147871, avg=147665.79, '
'stdev=146.58, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=45.68%, 50=52.75%, '
'100=1.53%\n'
' lat (usec) : 250=0.04%\n'
' cpu : usr=18.67%, '
'sys=55.16%, ctx=265038, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1033562,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=577MiB/s (605MB/s), '
'577MiB/s-577MiB/s '
'(605MB/s-605MB/s), io=4037MiB '
'(4233MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_213.output'),
'state': {'begin': 1666726912.9211314,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8747694492340088,
'end': 1666726913.7959008,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_213.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_213.state')},
'cmd_214': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_214.output'),
'state': {'begin': 1666726914.929846,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7863180637359619,
'end': 1666726915.716164,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_214.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_214.state')},
'cmd_215': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_215.output'),
'state': {'begin': 1666726915.718231,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.057596683502197,
'end': 1666726926.7758276,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_215.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_215.state')},
'cmd_216': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734016,\n'
' "timestamp_ms" : '
'1666734016263,\n'
' "time" : "Tue Oct 25 23:40:16 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'4226142208,\n'
' "io_kbytes" : 4127092,\n'
' "bw_bytes" : 603648365,\n'
' "bw" : 589500,\n'
' "iops" : 147374.517926,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1031769,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1993,\n'
' "max" : 26640,\n'
' "mean" : 2267.380921,\n'
' "stddev" : '
'434.200456,\n'
' "N" : 1031770\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 310,\n'
' "max" : 200466,\n'
' "mean" : '
'24390.233048,\n'
' "stddev" : '
'7872.725576,\n'
' "N" : 1031772,\n'
' "percentile" : {\n'
' "1.000000" : 16192,\n'
' "5.000000" : 16512,\n'
' "10.000000" : '
'16768,\n'
' "20.000000" : '
'17280,\n'
' "30.000000" : '
'17536,\n'
' "40.000000" : '
'18304,\n'
' "50.000000" : '
'28288,\n'
' "60.000000" : '
'28544,\n'
' "70.000000" : '
'28800,\n'
' "80.000000" : '
'29056,\n'
' "90.000000" : '
'29568,\n'
' "95.000000" : '
'34048,\n'
' "99.000000" : '
'51968,\n'
' "99.500000" : '
'53504,\n'
' "99.900000" : '
'84480,\n'
' "99.950000" : '
'99840,\n'
' "99.990000" : '
'115200\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 13556,\n'
' "max" : 203111,\n'
' "mean" : '
'26657.614235,\n'
' "stddev" : '
'7990.434757,\n'
' "N" : 1031772\n'
' },\n'
' "bw_min" : 587911,\n'
' "bw_max" : 591096,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'589642.214286,\n'
' "bw_dev" : 928.622982,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 146977,\n'
' "iops_max" : 147774,\n'
' "iops_mean" : '
'147410.500000,\n'
' "iops_stddev" : '
'232.283232,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 18.514286,\n'
' "sys_cpu" : 56.057143,\n'
' "ctx" : 264480,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 45.311111,\n'
' "50" : 53.090178,\n'
' "100" : 1.546858,\n'
' "250" : 0.049139,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=87473: Tue '
'Oct 25 23:40:16 2022\n'
' read: IOPS=147k, BW=576MiB/s '
'(604MB/s)(4030MiB/7001msec)\n'
' slat (nsec): min=1993, '
'max=26640, avg=2267.38, '
'stdev=434.20\n'
' clat (nsec): min=310, '
'max=200466, avg=24390.23, '
'stdev=7872.73\n'
' lat (usec): min=13, '
'max=203, avg=26.66, stdev= 7.99\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 17], '
'5.00th=[ 17], 10.00th=[ 17], '
'20.00th=[ 18],\n'
' | 30.00th=[ 18], '
'40.00th=[ 19], 50.00th=[ '
'29], 60.00th=[ 29],\n'
' | 70.00th=[ 29], '
'80.00th=[ 30], 90.00th=[ '
'30], 95.00th=[ 35],\n'
' | 99.00th=[ 52], '
'99.50th=[ 54], 99.90th=[ '
'85], 99.95th=[ 100],\n'
' | 99.99th=[ 116]\n'
' bw ( KiB/s): min=587911, '
'max=591096, per=100.00%, '
'avg=589642.21, stdev=928.62, '
'samples=14\n'
' iops : min=146977, '
'max=147774, avg=147410.50, '
'stdev=232.28, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=45.31%, 50=53.09%, '
'100=1.55%\n'
' lat (usec) : 250=0.05%\n'
' cpu : usr=18.51%, '
'sys=56.06%, ctx=264480, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1031769,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=576MiB/s (604MB/s), '
'576MiB/s-576MiB/s '
'(604MB/s-604MB/s), io=4030MiB '
'(4226MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_216.output'),
'state': {'begin': 1666726926.7787688,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7961454391479492,
'end': 1666726927.5749142,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_216.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_216.state')},
'cmd_217': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_217.output'),
'state': {'begin': 1666726929.6196554,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7881889343261719,
'end': 1666726930.4078443,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_217.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_217.state')},
'cmd_218': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_218.output'),
'state': {'begin': 1666726930.4099026,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.354801654815674,
'end': 1666726941.7647042,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_218.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_218.state')},
'cmd_219': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734031,\n'
' "timestamp_ms" : '
'1666734031180,\n'
' "time" : "Tue Oct 25 23:40:31 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'5484310528,\n'
' "io_kbytes" : 5355772,\n'
' "bw_bytes" : 783361023,\n'
' "bw" : 765000,\n'
' "iops" : 191249.821454,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1338940,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 661,\n'
' "max" : 38161,\n'
' "mean" : 943.770155,\n'
' "stddev" : '
'157.273061,\n'
' "N" : 1338940\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1282,\n'
' "max" : 194796,\n'
' "mean" : '
'19434.049774,\n'
' "stddev" : '
'6553.654058,\n'
' "N" : 1338942,\n'
' "percentile" : {\n'
' "1.000000" : 11200,\n'
' "5.000000" : 12480,\n'
' "10.000000" : '
'12992,\n'
' "20.000000" : '
'15936,\n'
' "30.000000" : '
'16320,\n'
' "40.000000" : '
'17024,\n'
' "50.000000" : '
'18048,\n'
' "60.000000" : '
'19328,\n'
' "70.000000" : '
'20352,\n'
' "80.000000" : '
'22656,\n'
' "90.000000" : '
'24448,\n'
' "95.000000" : '
'32640,\n'
' "99.000000" : '
'44288,\n'
' "99.500000" : '
'47360,\n'
' "99.900000" : '
'77312,\n'
' "99.950000" : '
'92672,\n'
' "99.990000" : '
'105984\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11071,\n'
' "max" : 195707,\n'
' "mean" : '
'20377.819873,\n'
' "stddev" : '
'6563.731917,\n'
' "N" : 1338942\n'
' },\n'
' "bw_min" : 764144,\n'
' "bw_max" : 766368,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'765234.071429,\n'
' "bw_dev" : 634.098083,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 191036,\n'
' "iops_max" : 191592,\n'
' "iops_mean" : '
'191308.357143,\n'
' "iops_stddev" : '
'158.356757,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 9,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 19.636429,\n'
' "4" : 80.350725,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 65.775315,\n'
' "50" : 33.972023,\n'
' "100" : 0.223236,\n'
' "250" : 0.027634,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=87522: Tue Oct 25 23:40:31 '
'2022\n'
' read: IOPS=191k, BW=747MiB/s '
'(783MB/s)(5230MiB/7001msec)\n'
' slat (nsec): min=661, '
'max=38161, avg=943.77, '
'stdev=157.27\n'
' clat (nsec): min=1282, '
'max=194796, avg=19434.05, '
'stdev=6553.65\n'
' lat (usec): min=11, '
'max=195, avg=20.38, stdev= 6.56\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 12], '
'5.00th=[ 13], 10.00th=[ 13], '
'20.00th=[ 16],\n'
' | 30.00th=[ 17], '
'40.00th=[ 18], 50.00th=[ '
'19], 60.00th=[ 20],\n'
' | 70.00th=[ 21], '
'80.00th=[ 23], 90.00th=[ '
'25], 95.00th=[ 33],\n'
' | 99.00th=[ 45], '
'99.50th=[ 48], 99.90th=[ '
'78], 99.95th=[ 93],\n'
' | 99.99th=[ 106]\n'
' bw ( KiB/s): min=764144, '
'max=766368, per=100.00%, '
'avg=765234.07, stdev=634.10, '
'samples=14\n'
' iops : min=191036, '
'max=191592, avg=191308.36, '
'stdev=158.36, samples=14\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.01%, 20=65.78%, '
'50=33.97%\n'
' lat (usec) : 100=0.22%, '
'250=0.03%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=9, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=19.6%, 4=80.4%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1338940,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=747MiB/s (783MB/s), '
'747MiB/s-747MiB/s '
'(783MB/s-783MB/s), io=5230MiB '
'(5484MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_219.output'),
'state': {'begin': 1666726941.7674084,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7939178943634033,
'end': 1666726942.5613263,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_219.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_219.state')},
'cmd_22': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_22.output'),
'state': {'begin': 1666725999.3979034,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7901809215545654,
'end': 1666726000.1880844,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_22.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_22.state')},
'cmd_220': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_220.output'),
'state': {'begin': 1666726944.5957801,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7897424697875977,
'end': 1666726945.3855226,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_220.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_220.state')},
'cmd_221': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_221.output'),
'state': {'begin': 1666726945.3878367,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.472328424453735,
'end': 1666726956.860165,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_221.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_221.state')},
'cmd_222': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734046,\n'
' "timestamp_ms" : '
'1666734046275,\n'
' "time" : "Tue Oct 25 23:40:46 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'5460344832,\n'
' "io_kbytes" : 5332368,\n'
' "bw_bytes" : 779937842,\n'
' "bw" : 761658,\n'
' "iops" : 190414.083702,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1333089,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 651,\n'
' "max" : 38282,\n'
' "mean" : 939.539594,\n'
' "stddev" : '
'143.340985,\n'
' "N" : 1333089\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1283,\n'
' "max" : 151294,\n'
' "mean" : '
'19526.900070,\n'
' "stddev" : '
'6576.455932,\n'
' "N" : 1333092,\n'
' "percentile" : {\n'
' "1.000000" : 11328,\n'
' "5.000000" : 12480,\n'
' "10.000000" : '
'13120,\n'
' "20.000000" : '
'16064,\n'
' "30.000000" : '
'16512,\n'
' "40.000000" : '
'17024,\n'
' "50.000000" : '
'18048,\n'
' "60.000000" : '
'19328,\n'
' "70.000000" : '
'20352,\n'
' "80.000000" : '
'22912,\n'
' "90.000000" : '
'24448,\n'
' "95.000000" : '
'32640,\n'
' "99.000000" : '
'44288,\n'
' "99.500000" : '
'47872,\n'
' "99.900000" : '
'79360,\n'
' "99.950000" : '
'93696,\n'
' "99.990000" : '
'105984\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10961,\n'
' "max" : 152466,\n'
' "mean" : '
'20466.439579,\n'
' "stddev" : '
'6586.533800,\n'
' "N" : 1333092\n'
' },\n'
' "bw_min" : 760152,\n'
' "bw_max" : 764288,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'761844.357143,\n'
' "bw_dev" : 972.547456,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 190038,\n'
' "iops_max" : 191072,\n'
' "iops_mean" : '
'190461.071429,\n'
' "iops_stddev" : '
'243.120883,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 19,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 19.228874,\n'
' "4" : 80.757024,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 65.134736,\n'
' "50" : 34.590939,\n'
' "100" : 0.244395,\n'
' "250" : 0.028730,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=87573: Tue Oct 25 23:40:46 '
'2022\n'
' read: IOPS=190k, BW=744MiB/s '
'(780MB/s)(5207MiB/7001msec)\n'
' slat (nsec): min=651, '
'max=38282, avg=939.54, '
'stdev=143.34\n'
' clat (nsec): min=1283, '
'max=151294, avg=19526.90, '
'stdev=6576.46\n'
' lat (usec): min=10, '
'max=152, avg=20.47, stdev= 6.59\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 12], '
'5.00th=[ 13], 10.00th=[ 14], '
'20.00th=[ 17],\n'
' | 30.00th=[ 17], '
'40.00th=[ 18], 50.00th=[ '
'19], 60.00th=[ 20],\n'
' | 70.00th=[ 21], '
'80.00th=[ 23], 90.00th=[ '
'25], 95.00th=[ 33],\n'
' | 99.00th=[ 45], '
'99.50th=[ 48], 99.90th=[ '
'80], 99.95th=[ 94],\n'
' | 99.99th=[ 106]\n'
' bw ( KiB/s): min=760152, '
'max=764288, per=100.00%, '
'avg=761844.36, stdev=972.55, '
'samples=14\n'
' iops : min=190038, '
'max=191072, avg=190461.07, '
'stdev=243.12, samples=14\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.01%, 20=65.13%, '
'50=34.59%\n'
' lat (usec) : 100=0.24%, '
'250=0.03%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=19, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=19.2%, 4=80.8%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1333089,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=744MiB/s (780MB/s), '
'744MiB/s-744MiB/s '
'(780MB/s-780MB/s), io=5207MiB '
'(5460MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_222.output'),
'state': {'begin': 1666726956.862852,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.782865047454834,
'end': 1666726957.6457171,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_222.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_222.state')},
'cmd_223': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_223.output'),
'state': {'begin': 1666726959.6549582,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7848594188690186,
'end': 1666726960.4398177,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_223.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_223.state')},
'cmd_224': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_224.output'),
'state': {'begin': 1666726960.4417746,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.47867488861084,
'end': 1666726971.9204495,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_224.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_224.state')},
'cmd_225': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734061,\n'
' "timestamp_ms" : '
'1666734061334,\n'
' "time" : "Tue Oct 25 23:41:01 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'5464141824,\n'
' "io_kbytes" : 5336076,\n'
' "bw_bytes" : 780480191,\n'
' "bw" : 762187,\n'
' "iops" : 190546.493358,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1334016,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 661,\n'
' "max" : 14577,\n'
' "mean" : 940.799248,\n'
' "stddev" : 97.471830,\n'
' "N" : 1334016\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 9919,\n'
' "max" : 193162,\n'
' "mean" : '
'19508.399007,\n'
' "stddev" : '
'6583.473141,\n'
' "N" : 1334019,\n'
' "percentile" : {\n'
' "1.000000" : 11328,\n'
' "5.000000" : 12480,\n'
' "10.000000" : '
'13120,\n'
' "20.000000" : '
'15936,\n'
' "30.000000" : '
'16512,\n'
' "40.000000" : '
'17024,\n'
' "50.000000" : '
'18048,\n'
' "60.000000" : '
'19328,\n'
' "70.000000" : '
'20352,\n'
' "80.000000" : '
'22912,\n'
' "90.000000" : '
'24448,\n'
' "95.000000" : '
'32640,\n'
' "99.000000" : '
'44288,\n'
' "99.500000" : '
'47872,\n'
' "99.900000" : '
'79360,\n'
' "99.950000" : '
'92672,\n'
' "99.990000" : '
'107008\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10850,\n'
' "max" : 193833,\n'
' "mean" : '
'20449.198191,\n'
' "stddev" : '
'6592.555887,\n'
' "N" : 1334019\n'
' },\n'
' "bw_min" : 760784,\n'
' "bw_max" : 763832,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'762373.571429,\n'
' "bw_dev" : 891.857932,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 190196,\n'
' "iops_max" : 190958,\n'
' "iops_mean" : '
'190593.357143,\n'
' "iops_stddev" : '
'222.929312,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 18,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 19.315660,\n'
' "4" : 80.683590,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 65.097870,\n'
' "50" : 34.639989,\n'
' "100" : 0.232681,\n'
' "250" : 0.029610,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=87623: Tue Oct 25 23:41:01 '
'2022\n'
' read: IOPS=191k, BW=744MiB/s '
'(780MB/s)(5211MiB/7001msec)\n'
' slat (nsec): min=661, '
'max=14577, avg=940.80, '
'stdev=97.47\n'
' clat (usec): min=9, max=193, '
'avg=19.51, stdev= 6.58\n'
' lat (usec): min=10, '
'max=193, avg=20.45, stdev= 6.59\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 12], '
'5.00th=[ 13], 10.00th=[ 14], '
'20.00th=[ 16],\n'
' | 30.00th=[ 17], '
'40.00th=[ 18], 50.00th=[ '
'19], 60.00th=[ 20],\n'
' | 70.00th=[ 21], '
'80.00th=[ 23], 90.00th=[ '
'25], 95.00th=[ 33],\n'
' | 99.00th=[ 45], '
'99.50th=[ 48], 99.90th=[ '
'80], 99.95th=[ 93],\n'
' | 99.99th=[ 108]\n'
' bw ( KiB/s): min=760784, '
'max=763832, per=100.00%, '
'avg=762373.57, stdev=891.86, '
'samples=14\n'
' iops : min=190196, '
'max=190958, avg=190593.36, '
'stdev=222.93, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=65.10%, 50=34.64%, 100=0.23%, '
'250=0.03%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=18, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=19.3%, 4=80.7%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1334016,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=744MiB/s (780MB/s), '
'744MiB/s-744MiB/s '
'(780MB/s-780MB/s), io=5211MiB '
'(5464MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_225.output'),
'state': {'begin': 1666726971.9233382,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7887589931488037,
'end': 1666726972.7120972,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_225.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_225.state')},
'cmd_226': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_226.output'),
'state': {'begin': 1666726973.7527087,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.784700870513916,
'end': 1666726974.5374095,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_226.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_226.state')},
'cmd_227': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_227.output'),
'state': {'begin': 1666726974.539415,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring-reference_0.txt"',
'cwd': 'None',
'elapsed': 11.03250789642334,
'end': 1666726985.5719228,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_227.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_227.state')},
'cmd_228': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=io_uring, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734075,\n'
' "timestamp_ms" : '
'1666734075058,\n'
' "time" : "Tue Oct 25 23:41:15 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'4150661120,\n'
' "io_kbytes" : 4053380,\n'
' "bw_bytes" : 592866893,\n'
' "bw" : 578971,\n'
' "iops" : 144742.465362,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1013342,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2234,\n'
' "max" : 35627,\n'
' "mean" : 2427.005221,\n'
' "stddev" : '
'421.685182,\n'
' "N" : 1013342\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 160,\n'
' "max" : 159249,\n'
' "mean" : '
'24692.870506,\n'
' "stddev" : '
'8160.319348,\n'
' "N" : 1013345,\n'
' "percentile" : {\n'
' "1.000000" : 14400,\n'
' "5.000000" : 14528,\n'
' "10.000000" : '
'14912,\n'
' "20.000000" : '
'16768,\n'
' "30.000000" : '
'17536,\n'
' "40.000000" : '
'20864,\n'
' "50.000000" : '
'28800,\n'
' "60.000000" : '
'29056,\n'
' "70.000000" : '
'29312,\n'
' "80.000000" : '
'29824,\n'
' "90.000000" : '
'30080,\n'
' "95.000000" : '
'34560,\n'
' "99.000000" : '
'52992,\n'
' "99.500000" : '
'55552,\n'
' "99.900000" : '
'80384,\n'
' "99.950000" : '
'96768,\n'
' "99.990000" : '
'113152\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 13666,\n'
' "max" : 161533,\n'
' "mean" : '
'27119.875680,\n'
' "stddev" : '
'8241.903875,\n'
' "N" : 1013345\n'
' },\n'
' "bw_min" : 577330,\n'
' "bw_max" : 581224,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'579190.428571,\n'
' "bw_dev" : 1221.195424,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 144332,\n'
' "iops_max" : 145306,\n'
' "iops_mean" : '
'144797.571429,\n'
' "iops_stddev" : '
'305.357474,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.071429,\n'
' "sys_cpu" : 60.985714,\n'
' "ctx" : 259323,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 35.179239,\n'
' "50" : 63.216762,\n'
' "100" : 1.556039,\n'
' "250" : 0.043223,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 1452241,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 21892,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 21892,\n'
' "util" : 99.049128\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=87662: Tue '
'Oct 25 23:41:15 2022\n'
' read: IOPS=145k, BW=565MiB/s '
'(593MB/s)(3958MiB/7001msec)\n'
' slat (nsec): min=2234, '
'max=35627, avg=2427.01, '
'stdev=421.69\n'
' clat (nsec): min=160, '
'max=159249, avg=24692.87, '
'stdev=8160.32\n'
' lat (usec): min=13, '
'max=161, avg=27.12, stdev= 8.24\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 15], '
'5.00th=[ 15], 10.00th=[ 15], '
'20.00th=[ 17],\n'
' | 30.00th=[ 18], '
'40.00th=[ 21], 50.00th=[ '
'29], 60.00th=[ 30],\n'
' | 70.00th=[ 30], '
'80.00th=[ 30], 90.00th=[ '
'31], 95.00th=[ 35],\n'
' | 99.00th=[ 53], '
'99.50th=[ 56], 99.90th=[ '
'81], 99.95th=[ 97],\n'
' | 99.99th=[ 114]\n'
' bw ( KiB/s): min=577330, '
'max=581224, per=100.00%, '
'avg=579190.43, stdev=1221.20, '
'samples=14\n'
' iops : min=144332, '
'max=145306, avg=144797.57, '
'stdev=305.36, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=35.18%, 50=63.22%, '
'100=1.56%\n'
' lat (usec) : 250=0.04%\n'
' cpu : usr=15.07%, '
'sys=60.99%, ctx=259323, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1013342,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=565MiB/s (593MB/s), '
'565MiB/s-565MiB/s '
'(593MB/s-593MB/s), io=3958MiB '
'(4151MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=1452241/0, '
'merge=0/0, ticks=21892/0, '
'in_queue=21892, util=99.05%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_228.output'),
'state': {'begin': 1666726985.574558,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7973747253417969,
'end': 1666726986.3719327,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_228.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_228.state')},
'cmd_229': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_229.output'),
'state': {'begin': 1666726987.367063,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7926807403564453,
'end': 1666726988.1597438,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_229.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_229.state')},
'cmd_23': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_23.output'),
'state': {'begin': 1666726000.1902053,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.493699789047241,
'end': 1666726011.6839051,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_23.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_23.state')},
'cmd_230': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_230.output'),
'state': {'begin': 1666726988.1618998,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring-reference_1.txt"',
'cwd': 'None',
'elapsed': 11.053723812103271,
'end': 1666726999.2156236,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_230.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_230.state')},
'cmd_231': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=io_uring, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734088,\n'
' "timestamp_ms" : '
'1666734088701,\n'
' "time" : "Tue Oct 25 23:41:28 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'4203417600,\n'
' "io_kbytes" : 4104900,\n'
' "bw_bytes" : 600402456,\n'
' "bw" : 586330,\n'
' "iops" : 146582.202542,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1026222,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2174,\n'
' "max" : 178164,\n'
' "mean" : 2386.173347,\n'
' "stddev" : '
'455.786112,\n'
' "N" : 1026222\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 191,\n'
' "max" : 208732,\n'
' "mean" : '
'24370.688623,\n'
' "stddev" : '
'8077.609207,\n'
' "N" : 1026225,\n'
' "percentile" : {\n'
' "1.000000" : 14272,\n'
' "5.000000" : 15040,\n'
' "10.000000" : '
'16512,\n'
' "20.000000" : '
'16768,\n'
' "30.000000" : '
'17280,\n'
' "40.000000" : '
'17792,\n'
' "50.000000" : '
'28544,\n'
' "60.000000" : '
'28800,\n'
' "70.000000" : '
'28800,\n'
' "80.000000" : '
'29056,\n'
' "90.000000" : '
'29568,\n'
' "95.000000" : '
'34560,\n'
' "99.000000" : '
'52480,\n'
' "99.500000" : '
'54016,\n'
' "99.900000" : '
'83456,\n'
' "99.950000" : '
'99840,\n'
' "99.990000" : '
'115200\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11301,\n'
' "max" : 217257,\n'
' "mean" : '
'26756.861984,\n'
' "stddev" : '
'8171.492717,\n'
' "N" : 1026225\n'
' },\n'
' "bw_min" : 584704,\n'
' "bw_max" : 588312,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'586556.714286,\n'
' "bw_dev" : 891.194395,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 146176,\n'
' "iops_max" : 147078,\n'
' "iops_mean" : '
'146639.000000,\n'
' "iops_stddev" : '
'222.618681,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.600000,\n'
' "sys_cpu" : 60.242857,\n'
' "ctx" : 262841,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 41.038976,\n'
' "50" : 57.387485,\n'
' "100" : 1.520529,\n'
' "250" : 0.049210,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 1469749,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 22115,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 22115,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=87701: Tue '
'Oct 25 23:41:28 2022\n'
' read: IOPS=147k, BW=573MiB/s '
'(600MB/s)(4009MiB/7001msec)\n'
' slat (nsec): min=2174, '
'max=178164, avg=2386.17, '
'stdev=455.79\n'
' clat (nsec): min=191, '
'max=208732, avg=24370.69, '
'stdev=8077.61\n'
' lat (usec): min=11, '
'max=217, avg=26.76, stdev= 8.17\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 15], '
'5.00th=[ 16], 10.00th=[ 17], '
'20.00th=[ 17],\n'
' | 30.00th=[ 18], '
'40.00th=[ 18], 50.00th=[ '
'29], 60.00th=[ 29],\n'
' | 70.00th=[ 29], '
'80.00th=[ 30], 90.00th=[ '
'30], 95.00th=[ 35],\n'
' | 99.00th=[ 53], '
'99.50th=[ 55], 99.90th=[ '
'84], 99.95th=[ 100],\n'
' | 99.99th=[ 116]\n'
' bw ( KiB/s): min=584704, '
'max=588312, per=100.00%, '
'avg=586556.71, stdev=891.19, '
'samples=14\n'
' iops : min=146176, '
'max=147078, avg=146639.00, '
'stdev=222.62, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=41.04%, 50=57.39%, '
'100=1.52%\n'
' lat (usec) : 250=0.05%\n'
' cpu : usr=15.60%, '
'sys=60.24%, ctx=262841, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1026222,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=573MiB/s (600MB/s), '
'573MiB/s-573MiB/s '
'(600MB/s-600MB/s), io=4009MiB '
'(4203MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=1469749/0, '
'merge=0/0, ticks=22115/0, '
'in_queue=22115, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_231.output'),
'state': {'begin': 1666726999.218482,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7940006256103516,
'end': 1666727000.0124826,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_231.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_231.state')},
'cmd_232': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_232.output'),
'state': {'begin': 1666727001.0083597,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7922351360321045,
'end': 1666727001.8005948,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_232.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_232.state')},
'cmd_233': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_233.output'),
'state': {'begin': 1666727001.8026242,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring-reference_2.txt"',
'cwd': 'None',
'elapsed': 11.044522523880005,
'end': 1666727012.8471467,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_233.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_233.state')},
'cmd_234': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=io_uring, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734102,\n'
' "timestamp_ms" : '
'1666734102332,\n'
' "time" : "Tue Oct 25 23:41:42 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'4528324608,\n'
' "io_kbytes" : 4422192,\n'
' "bw_bytes" : 646811113,\n'
' "bw" : 631651,\n'
' "iops" : 157912.441080,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1105545,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2275,\n'
' "max" : 119033,\n'
' "mean" : 2745.671780,\n'
' "stddev" : '
'595.147245,\n'
' "N" : 1105545\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 3747,\n'
' "max" : 138570,\n'
' "mean" : '
'22051.316446,\n'
' "stddev" : '
'5281.925427,\n'
' "N" : 1105548,\n'
' "percentile" : {\n'
' "1.000000" : 17792,\n'
' "5.000000" : 19328,\n'
' "10.000000" : '
'19328,\n'
' "20.000000" : '
'19840,\n'
' "30.000000" : '
'20864,\n'
' "40.000000" : '
'20864,\n'
' "50.000000" : '
'21120,\n'
' "60.000000" : '
'21120,\n'
' "70.000000" : '
'21376,\n'
' "80.000000" : '
'21632,\n'
' "90.000000" : '
'22400,\n'
' "95.000000" : '
'33024,\n'
' "99.000000" : '
'47360,\n'
' "99.500000" : '
'50432,\n'
' "99.900000" : '
'79360,\n'
' "99.950000" : '
'93696,\n'
' "99.990000" : '
'105984\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12774,\n'
' "max" : 140935,\n'
' "mean" : '
'24796.987255,\n'
' "stddev" : '
'5227.303506,\n'
' "N" : 1105548\n'
' },\n'
' "bw_min" : 630016,\n'
' "bw_max" : 633688,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'632033.142857,\n'
' "bw_dev" : 1130.843651,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 157504,\n'
' "iops_max" : 158422,\n'
' "iops_mean" : '
'158008.214286,\n'
' "iops_stddev" : '
'282.708923,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.085714,\n'
' "sys_cpu" : 57.100000,\n'
' "ctx" : 284692,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 21.244273,\n'
' "50" : 78.183339,\n'
' "100" : 0.536839,\n'
' "250" : 0.034915,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 1564440,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 23344,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 23344,\n'
' "util" : 99.108381\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=87740: Tue '
'Oct 25 23:41:42 2022\n'
' read: IOPS=158k, BW=617MiB/s '
'(647MB/s)(4319MiB/7001msec)\n'
' slat (nsec): min=2275, '
'max=119033, avg=2745.67, '
'stdev=595.15\n'
' clat (usec): min=3, max=138, '
'avg=22.05, stdev= 5.28\n'
' lat (usec): min=12, '
'max=140, avg=24.80, stdev= 5.23\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 18], '
'5.00th=[ 20], 10.00th=[ 20], '
'20.00th=[ 20],\n'
' | 30.00th=[ 21], '
'40.00th=[ 21], 50.00th=[ '
'22], 60.00th=[ 22],\n'
' | 70.00th=[ 22], '
'80.00th=[ 22], 90.00th=[ '
'23], 95.00th=[ 34],\n'
' | 99.00th=[ 48], '
'99.50th=[ 51], 99.90th=[ '
'80], 99.95th=[ 94],\n'
' | 99.99th=[ 106]\n'
' bw ( KiB/s): min=630016, '
'max=633688, per=100.00%, '
'avg=632033.14, stdev=1130.84, '
'samples=14\n'
' iops : min=157504, '
'max=158422, avg=158008.21, '
'stdev=282.71, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=21.24%, 50=78.18%, '
'100=0.54%\n'
' lat (usec) : 250=0.03%\n'
' cpu : usr=14.09%, '
'sys=57.10%, ctx=284692, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1105545,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=617MiB/s (647MB/s), '
'617MiB/s-617MiB/s '
'(647MB/s-647MB/s), io=4319MiB '
'(4528MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=1564440/0, '
'merge=0/0, ticks=23344/0, '
'in_queue=23344, util=99.11%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_234.output'),
'state': {'begin': 1666727012.849836,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7898743152618408,
'end': 1666727013.6397104,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_234.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_234.state')},
'cmd_235': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_235.output'),
'state': {'begin': 1666727015.6616108,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.7738947868347168,
'end': 1666727016.4355056,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_235.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_235.state')},
'cmd_236': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_236.output'),
'state': {'begin': 1666727016.437498,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt"',
'cwd': 'None',
'elapsed': 11.448853969573975,
'end': 1666727027.886352,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_236.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_236.state')},
'cmd_237': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734117,\n'
' "timestamp_ms" : '
'1666734117300,\n'
' "time" : "Tue Oct 25 23:41:57 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'5844619264,\n'
' "io_kbytes" : 5707636,\n'
' "bw_bytes" : 834826348,\n'
' "bw" : 815260,\n'
' "iops" : 203814.597915,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1426906,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 15208,\n'
' "mean" : 166.282201,\n'
' "stddev" : 53.441501,\n'
' "N" : 1426906\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 10440,\n'
' "max" : 145372,\n'
' "mean" : '
'19068.696730,\n'
' "stddev" : '
'5673.395713,\n'
' "N" : 1426908,\n'
' "percentile" : {\n'
' "1.000000" : 15424,\n'
' "5.000000" : 16320,\n'
' "10.000000" : '
'17024,\n'
' "20.000000" : '
'17536,\n'
' "30.000000" : '
'17536,\n'
' "40.000000" : '
'17536,\n'
' "50.000000" : '
'17792,\n'
' "60.000000" : '
'17792,\n'
' "70.000000" : '
'18048,\n'
' "80.000000" : '
'18048,\n'
' "90.000000" : '
'19584,\n'
' "95.000000" : '
'31104,\n'
' "99.000000" : '
'43776,\n'
' "99.500000" : '
'46336,\n'
' "99.900000" : '
'83456,\n'
' "99.950000" : '
'98816,\n'
' "99.990000" : '
'123392\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10620,\n'
' "max" : 145553,\n'
' "mean" : '
'19234.978923,\n'
' "stddev" : '
'5673.110075,\n'
' "N" : 1426908\n'
' },\n'
' "bw_min" : 812432,\n'
' "bw_max" : 821851,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'815661.571429,\n'
' "bw_dev" : 2674.021567,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 203108,\n'
' "iops_max" : 205462,\n'
' "iops_mean" : '
'203915.428571,\n'
' "iops_stddev" : '
'668.389306,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 60.214286,\n'
' "sys_cpu" : 39.757143,\n'
' "ctx" : 14,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 30.973379,\n'
' "4" : 69.014077,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 91.161156,\n'
' "50" : 8.538124,\n'
' "100" : 0.253556,\n'
' "250" : 0.047305,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=87788: Tue '
'Oct 25 23:41:57 2022\n'
' read: IOPS=204k, BW=796MiB/s '
'(835MB/s)(5574MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=15208, avg=166.28, '
'stdev=53.44\n'
' clat (usec): min=10, '
'max=145, avg=19.07, stdev= 5.67\n'
' lat (usec): min=10, '
'max=145, avg=19.23, stdev= 5.67\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 16], '
'5.00th=[ 17], 10.00th=[ 18], '
'20.00th=[ 18],\n'
' | 30.00th=[ 18], '
'40.00th=[ 18], 50.00th=[ '
'18], 60.00th=[ 18],\n'
' | 70.00th=[ 19], '
'80.00th=[ 19], 90.00th=[ '
'20], 95.00th=[ 32],\n'
' | 99.00th=[ 44], '
'99.50th=[ 47], 99.90th=[ '
'84], 99.95th=[ 99],\n'
' | 99.99th=[ 124]\n'
' bw ( KiB/s): min=812432, '
'max=821851, per=100.00%, '
'avg=815661.57, stdev=2674.02, '
'samples=14\n'
' iops : min=203108, '
'max=205462, avg=203915.43, '
'stdev=668.39, samples=14\n'
' lat (usec) : 20=91.16%, '
'50=8.54%, 100=0.25%, 250=0.05%\n'
' cpu : usr=60.21%, '
'sys=39.76%, ctx=14, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=31.0%, 4=69.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1426906,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=796MiB/s (835MB/s), '
'796MiB/s-796MiB/s '
'(835MB/s-835MB/s), io=5574MiB '
'(5845MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_237.output'),
'state': {'begin': 1666727027.887692,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.7917938232421875,
'end': 1666727028.6794858,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_237.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_237.state')},
'cmd_238': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_238.output'),
'state': {'begin': 1666727030.7304463,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7934126853942871,
'end': 1666727031.523859,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_238.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_238.state')},
'cmd_239': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_239.output'),
'state': {'begin': 1666727031.5263135,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt"',
'cwd': 'None',
'elapsed': 11.473499536514282,
'end': 1666727042.999813,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_239.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_239.state')},
'cmd_24': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733101,\n'
' "timestamp_ms" : '
'1666733101115,\n'
' "time" : "Tue Oct 25 23:25:01 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2411470848,\n'
' "io_kbytes" : 2354952,\n'
' "bw_bytes" : 344446628,\n'
' "bw" : 336373,\n'
' "iops" : 84093.415226,\n'
' "runtime" : 7001,\n'
' "total_ios" : 588738,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 711,\n'
' "max" : 25067,\n'
' "mean" : 928.008393,\n'
' "stddev" : 146.987262,\n'
' "N" : 588738\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 701,\n'
' "max" : 113202,\n'
' "mean" : 10713.652358,\n'
' "stddev" : '
'2206.439842,\n'
' "N" : 588738,\n'
' "percentile" : {\n'
' "1.000000" : 9920,\n'
' "5.000000" : 10048,\n'
' "10.000000" : 10176,\n'
' "20.000000" : 10176,\n'
' "30.000000" : 10304,\n'
' "40.000000" : 10304,\n'
' "50.000000" : 10432,\n'
' "60.000000" : 10432,\n'
' "70.000000" : 10432,\n'
' "80.000000" : 10560,\n'
' "90.000000" : 10688,\n'
' "95.000000" : 11200,\n'
' "99.000000" : 20096,\n'
' "99.500000" : 27520,\n'
' "99.900000" : 30592,\n'
' "99.950000" : 43776,\n'
' "99.990000" : 67072\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10590,\n'
' "max" : 114104,\n'
' "mean" : 11641.660751,\n'
' "stddev" : '
'2210.286923,\n'
' "N" : 588738\n'
' },\n'
' "bw_min" : 335992,\n'
' "bw_max" : 336976,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'336465.214286,\n'
' "bw_dev" : 296.234361,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 83998,\n'
' "iops_max" : 84244,\n'
' "iops_mean" : '
'84116.285714,\n'
' "iops_stddev" : '
'74.034741,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 15,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 1.542282,\n'
' "20" : 97.451498,\n'
' "50" : 0.959000,\n'
' "100" : 0.044842,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=84683: Tue Oct 25 23:25:01 '
'2022\n'
' read: IOPS=84.1k, BW=328MiB/s '
'(344MB/s)(2300MiB/7001msec)\n'
' slat (nsec): min=711, '
'max=25067, avg=928.01, '
'stdev=146.99\n'
' clat (nsec): min=701, '
'max=113202, avg=10713.65, '
'stdev=2206.44\n'
' lat (usec): min=10, max=114, '
'avg=11.64, stdev= 2.21\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9920], '
'5.00th=[10048], 10.00th=[10176], '
'20.00th=[10176],\n'
' | 30.00th=[10304], '
'40.00th=[10304], 50.00th=[10432], '
'60.00th=[10432],\n'
' | 70.00th=[10432], '
'80.00th=[10560], 90.00th=[10688], '
'95.00th=[11200],\n'
' | 99.00th=[20096], '
'99.50th=[27520], 99.90th=[30592], '
'99.95th=[43776],\n'
' | 99.99th=[67072]\n'
' bw ( KiB/s): min=335992, '
'max=336976, per=100.00%, '
'avg=336465.21, stdev=296.23, '
'samples=14\n'
' iops : min=83998, '
'max=84244, avg=84116.29, '
'stdev=74.03, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=1.54%, 20=97.45%, '
'50=0.96%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=15, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=588738,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=328MiB/s (344MB/s), '
'328MiB/s-328MiB/s '
'(344MB/s-344MB/s), io=2300MiB '
'(2411MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_24.output'),
'state': {'begin': 1666726011.6866117,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7878735065460205,
'end': 1666726012.4744852,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_24.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_24.state')},
'cmd_240': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734132,\n'
' "timestamp_ms" : '
'1666734132411,\n'
' "time" : "Tue Oct 25 23:42:12 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6327066624,\n'
' "io_kbytes" : 6178776,\n'
' "bw_bytes" : 903737555,\n'
' "bw" : 882556,\n'
' "iops" : 220638.623054,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1544691,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 17432,\n'
' "mean" : 189.121560,\n'
' "stddev" : '
'121.269366,\n'
' "N" : 1544691\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 9367,\n'
' "max" : 202460,\n'
' "mean" : '
'17679.470528,\n'
' "stddev" : '
'5728.764872,\n'
' "N" : 1544694,\n'
' "percentile" : {\n'
' "1.000000" : 11072,\n'
' "5.000000" : 15296,\n'
' "10.000000" : '
'15936,\n'
' "20.000000" : '
'16064,\n'
' "30.000000" : '
'16064,\n'
' "40.000000" : '
'16064,\n'
' "50.000000" : '
'16192,\n'
' "60.000000" : '
'16192,\n'
' "70.000000" : '
'16320,\n'
' "80.000000" : '
'16512,\n'
' "90.000000" : '
'20608,\n'
' "95.000000" : '
'29312,\n'
' "99.000000" : '
'39168,\n'
' "99.500000" : '
'44288,\n'
' "99.900000" : '
'81408,\n'
' "99.950000" : '
'92672,\n'
' "99.990000" : '
'105984\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9678,\n'
' "max" : 202640,\n'
' "mean" : '
'17868.592084,\n'
' "stddev" : '
'5727.259130,\n'
' "N" : 1544694\n'
' },\n'
' "bw_min" : 880480,\n'
' "bw_max" : 884112,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'882771.142857,\n'
' "bw_dev" : 946.312914,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 220120,\n'
' "iops_max" : 221028,\n'
' "iops_mean" : '
'220692.928571,\n'
' "iops_stddev" : '
'236.341495,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 42.900000,\n'
' "sys_cpu" : 57.071429,\n'
' "ctx" : 11,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.363309,\n'
' "2" : 2.448580,\n'
' "4" : 97.188111,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 89.582124,\n'
' "50" : 10.162874,\n'
' "100" : 0.218814,\n'
' "250" : 0.032369,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=87838: Tue '
'Oct 25 23:42:12 2022\n'
' read: IOPS=221k, BW=862MiB/s '
'(904MB/s)(6034MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=17432, avg=189.12, '
'stdev=121.27\n'
' clat (usec): min=9, max=202, '
'avg=17.68, stdev= 5.73\n'
' lat (usec): min=9, max=202, '
'avg=17.87, stdev= 5.73\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 12], '
'5.00th=[ 16], 10.00th=[ 16], '
'20.00th=[ 17],\n'
' | 30.00th=[ 17], '
'40.00th=[ 17], 50.00th=[ '
'17], 60.00th=[ 17],\n'
' | 70.00th=[ 17], '
'80.00th=[ 17], 90.00th=[ '
'21], 95.00th=[ 30],\n'
' | 99.00th=[ 40], '
'99.50th=[ 45], 99.90th=[ '
'82], 99.95th=[ 93],\n'
' | 99.99th=[ 106]\n'
' bw ( KiB/s): min=880480, '
'max=884112, per=100.00%, '
'avg=882771.14, stdev=946.31, '
'samples=14\n'
' iops : min=220120, '
'max=221028, avg=220692.93, '
'stdev=236.34, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=89.58%, 50=10.16%, 100=0.22%, '
'250=0.03%\n'
' cpu : usr=42.90%, '
'sys=57.07%, ctx=11, majf=0, '
'minf=0\n'
' IO depths : 1=0.4%, 2=2.4%, '
'4=97.2%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1544691,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=862MiB/s (904MB/s), '
'862MiB/s-862MiB/s '
'(904MB/s-904MB/s), io=6034MiB '
'(6327MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_240.output'),
'state': {'begin': 1666727043.0025334,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7948098182678223,
'end': 1666727043.7973433,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_240.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_240.state')},
'cmd_241': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_241.output'),
'state': {'begin': 1666727045.8665073,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.7829034328460693,
'end': 1666727046.6494107,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_241.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_241.state')},
'cmd_242': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_242.output'),
'state': {'begin': 1666727046.6566803,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt"',
'cwd': 'None',
'elapsed': 11.461086511611938,
'end': 1666727058.1177669,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_242.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_242.state')},
'cmd_243': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734147,\n'
' "timestamp_ms" : '
'1666734147528,\n'
' "time" : "Tue Oct 25 23:42:27 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6299238400,\n'
' "io_kbytes" : 6151600,\n'
' "bw_bytes" : 899762662,\n'
' "bw" : 878674,\n'
' "iops" : 219668.190259,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1537897,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 120,\n'
' "max" : 16481,\n'
' "mean" : 214.052651,\n'
' "stddev" : '
'212.685902,\n'
' "N" : 1537897\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 9428,\n'
' "max" : 196909,\n'
' "mean" : '
'17698.782211,\n'
' "stddev" : '
'5860.452451,\n'
' "N" : 1537900,\n'
' "percentile" : {\n'
' "1.000000" : 11072,\n'
' "5.000000" : 14144,\n'
' "10.000000" : '
'15424,\n'
' "20.000000" : '
'15936,\n'
' "30.000000" : '
'16064,\n'
' "40.000000" : '
'16064,\n'
' "50.000000" : '
'16192,\n'
' "60.000000" : '
'16192,\n'
' "70.000000" : '
'16512,\n'
' "80.000000" : '
'17280,\n'
' "90.000000" : '
'21120,\n'
' "95.000000" : '
'29568,\n'
' "99.000000" : '
'41216,\n'
' "99.500000" : '
'44288,\n'
' "99.900000" : '
'81408,\n'
' "99.950000" : '
'93696,\n'
' "99.990000" : '
'105984\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9678,\n'
' "max" : 197099,\n'
' "mean" : '
'17912.834803,\n'
' "stddev" : '
'5855.839323,\n'
' "N" : 1537900\n'
' },\n'
' "bw_min" : 875880,\n'
' "bw_max" : 881400,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'878894.857143,\n'
' "bw_dev" : 1748.287903,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 218970,\n'
' "iops_max" : 220350,\n'
' "iops_mean" : '
'219723.714286,\n'
' "iops_stddev" : '
'437.071976,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 43.414286,\n'
' "sys_cpu" : 56.557143,\n'
' "ctx" : 12,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.328696,\n'
' "2" : 5.216539,\n'
' "4" : 94.454765,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 87.246285,\n'
' "50" : 12.507144,\n'
' "100" : 0.209442,\n'
' "250" : 0.031602,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=87889: Tue '
'Oct 25 23:42:27 2022\n'
' read: IOPS=220k, BW=858MiB/s '
'(900MB/s)(6007MiB/7001msec)\n'
' slat (nsec): min=120, '
'max=16481, avg=214.05, '
'stdev=212.69\n'
' clat (usec): min=9, max=196, '
'avg=17.70, stdev= 5.86\n'
' lat (usec): min=9, max=197, '
'avg=17.91, stdev= 5.86\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 12], '
'5.00th=[ 15], 10.00th=[ 16], '
'20.00th=[ 16],\n'
' | 30.00th=[ 17], '
'40.00th=[ 17], 50.00th=[ '
'17], 60.00th=[ 17],\n'
' | 70.00th=[ 17], '
'80.00th=[ 18], 90.00th=[ '
'22], 95.00th=[ 30],\n'
' | 99.00th=[ 42], '
'99.50th=[ 45], 99.90th=[ '
'82], 99.95th=[ 94],\n'
' | 99.99th=[ 106]\n'
' bw ( KiB/s): min=875880, '
'max=881400, per=100.00%, '
'avg=878894.86, stdev=1748.29, '
'samples=14\n'
' iops : min=218970, '
'max=220350, avg=219723.71, '
'stdev=437.07, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=87.25%, 50=12.51%, 100=0.21%, '
'250=0.03%\n'
' cpu : usr=43.41%, '
'sys=56.56%, ctx=12, majf=0, '
'minf=0\n'
' IO depths : 1=0.3%, 2=5.2%, '
'4=94.5%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1537897,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=858MiB/s (900MB/s), '
'858MiB/s-858MiB/s '
'(900MB/s-900MB/s), io=6007MiB '
'(6299MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_243.output'),
'state': {'begin': 1666727058.1210501,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.792534589767456,
'end': 1666727058.9135847,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_243.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_243.state')},
'cmd_244': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_244.output'),
'state': {'begin': 1666727060.973781,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7824304103851318,
'end': 1666727061.7562115,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_244.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_244.state')},
'cmd_245': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_245.output'),
'state': {'begin': 1666727061.7584105,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.355727434158325,
'end': 1666727073.114138,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_245.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_245.state')},
'cmd_246': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734162,\n'
' "timestamp_ms" : '
'1666734162525,\n'
' "time" : "Tue Oct 25 23:42:42 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6373732352,\n'
' "io_kbytes" : 6224348,\n'
' "bw_bytes" : 910403135,\n'
' "bw" : 889065,\n'
' "iops" : 222265.962005,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1556084,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1462,\n'
' "max" : 37179,\n'
' "mean" : 1641.605073,\n'
' "stddev" : '
'199.551002,\n'
' "N" : 1556084\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 230,\n'
' "max" : 126758,\n'
' "mean" : '
'16115.646010,\n'
' "stddev" : '
'5418.188609,\n'
' "N" : 1556087,\n'
' "percentile" : {\n'
' "1.000000" : 14144,\n'
' "5.000000" : 14400,\n'
' "10.000000" : '
'14528,\n'
' "20.000000" : '
'14528,\n'
' "30.000000" : '
'14656,\n'
' "40.000000" : '
'14656,\n'
' "50.000000" : '
'14656,\n'
' "60.000000" : '
'14784,\n'
' "70.000000" : '
'14784,\n'
' "80.000000" : '
'14912,\n'
' "90.000000" : '
'16512,\n'
' "95.000000" : '
'27008,\n'
' "99.000000" : '
'36096,\n'
' "99.500000" : '
'42752,\n'
' "99.900000" : '
'78336,\n'
' "99.950000" : '
'88576,\n'
' "99.990000" : '
'103936\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10790,\n'
' "max" : 128381,\n'
' "mean" : '
'17757.251002,\n'
' "stddev" : '
'5418.749748,\n'
' "N" : 1556087\n'
' },\n'
' "bw_min" : 888000,\n'
' "bw_max" : 890464,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'889269.357143,\n'
' "bw_dev" : 942.858143,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 222000,\n'
' "iops_max" : 222616,\n'
' "iops_mean" : '
'222317.285714,\n'
' "iops_stddev" : '
'235.776435,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 18.328571,\n'
' "sys_cpu" : 81.642857,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 91.798450,\n'
' "50" : 8.025659,\n'
' "100" : 0.146200,\n'
' "250" : 0.019986,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=87938: Tue '
'Oct 25 23:42:42 2022\n'
' read: IOPS=222k, BW=868MiB/s '
'(910MB/s)(6078MiB/7001msec)\n'
' slat (nsec): min=1462, '
'max=37179, avg=1641.61, '
'stdev=199.55\n'
' clat (nsec): min=230, '
'max=126758, avg=16115.65, '
'stdev=5418.19\n'
' lat (usec): min=10, '
'max=128, avg=17.76, stdev= 5.42\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 15], '
'5.00th=[ 15], 10.00th=[ 15], '
'20.00th=[ 15],\n'
' | 30.00th=[ 15], '
'40.00th=[ 15], 50.00th=[ '
'15], 60.00th=[ 15],\n'
' | 70.00th=[ 15], '
'80.00th=[ 15], 90.00th=[ '
'17], 95.00th=[ 28],\n'
' | 99.00th=[ 37], '
'99.50th=[ 43], 99.90th=[ '
'79], 99.95th=[ 89],\n'
' | 99.99th=[ 104]\n'
' bw ( KiB/s): min=888000, '
'max=890464, per=100.00%, '
'avg=889269.36, stdev=942.86, '
'samples=14\n'
' iops : min=222000, '
'max=222616, avg=222317.29, '
'stdev=235.78, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=91.80%, 50=8.03%, '
'100=0.15%\n'
' lat (usec) : 250=0.02%\n'
' cpu : usr=18.33%, '
'sys=81.64%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1556084,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=868MiB/s (910MB/s), '
'868MiB/s-868MiB/s '
'(910MB/s-910MB/s), io=6078MiB '
'(6374MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_246.output'),
'state': {'begin': 1666727073.1173422,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7967362403869629,
'end': 1666727073.9140785,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_246.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_246.state')},
'cmd_247': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_247.output'),
'state': {'begin': 1666727075.930548,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7862098217010498,
'end': 1666727076.7167578,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_247.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_247.state')},
'cmd_248': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_248.output'),
'state': {'begin': 1666727076.7187743,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.360539197921753,
'end': 1666727088.0793135,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_248.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_248.state')},
'cmd_249': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734177,\n'
' "timestamp_ms" : '
'1666734177491,\n'
' "time" : "Tue Oct 25 23:42:57 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6295175168,\n'
' "io_kbytes" : 6147632,\n'
' "bw_bytes" : 899182283,\n'
' "bw" : 878107,\n'
' "iops" : 219526.496215,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1536905,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1593,\n'
' "max" : 27100,\n'
' "mean" : 1862.124565,\n'
' "stddev" : '
'155.390262,\n'
' "N" : 1536905\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 250,\n'
' "max" : 140985,\n'
' "mean" : '
'16110.551588,\n'
' "stddev" : '
'6695.143158,\n'
' "N" : 1536908,\n'
' "percentile" : {\n'
' "1.000000" : 10432,\n'
' "5.000000" : 10432,\n'
' "10.000000" : '
'10560,\n'
' "20.000000" : '
'12480,\n'
' "30.000000" : '
'12736,\n'
' "40.000000" : '
'13248,\n'
' "50.000000" : '
'14400,\n'
' "60.000000" : '
'14528,\n'
' "70.000000" : '
'16768,\n'
' "80.000000" : '
'20608,\n'
' "90.000000" : '
'21376,\n'
' "95.000000" : '
'29312,\n'
' "99.000000" : '
'41216,\n'
' "99.500000" : '
'44800,\n'
' "99.900000" : '
'80384,\n'
' "99.950000" : '
'96768,\n'
' "99.990000" : '
'122368\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9629,\n'
' "max" : 142838,\n'
' "mean" : '
'17972.676267,\n'
' "stddev" : '
'6706.536732,\n'
' "N" : 1536908\n'
' },\n'
' "bw_min" : 876896,\n'
' "bw_max" : 880945,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'878672.428571,\n'
' "bw_dev" : 1464.370472,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 219224,\n'
' "iops_max" : 220236,\n'
' "iops_mean" : '
'219668.071429,\n'
' "iops_stddev" : '
'366.064405,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.700000,\n'
' "sys_cpu" : 82.271429,\n'
' "ctx" : 18,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.043334,\n'
' "20" : 76.322609,\n'
' "50" : 23.389604,\n'
' "100" : 0.200533,\n'
' "250" : 0.043919,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=87987: Tue '
'Oct 25 23:42:57 2022\n'
' read: IOPS=220k, BW=858MiB/s '
'(899MB/s)(6004MiB/7001msec)\n'
' slat (nsec): min=1593, '
'max=27100, avg=1862.12, '
'stdev=155.39\n'
' clat (nsec): min=250, '
'max=140985, avg=16110.55, '
'stdev=6695.14\n'
' lat (usec): min=9, max=142, '
'avg=17.97, stdev= 6.71\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 11], '
'5.00th=[ 11], 10.00th=[ 11], '
'20.00th=[ 13],\n'
' | 30.00th=[ 13], '
'40.00th=[ 14], 50.00th=[ '
'15], 60.00th=[ 15],\n'
' | 70.00th=[ 17], '
'80.00th=[ 21], 90.00th=[ '
'22], 95.00th=[ 30],\n'
' | 99.00th=[ 42], '
'99.50th=[ 45], 99.90th=[ '
'81], 99.95th=[ 97],\n'
' | 99.99th=[ 123]\n'
' bw ( KiB/s): min=876896, '
'max=880945, per=100.00%, '
'avg=878672.43, stdev=1464.37, '
'samples=14\n'
' iops : min=219224, '
'max=220236, avg=219668.07, '
'stdev=366.06, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 10=0.04%, '
'20=76.32%, 50=23.39%, 100=0.20%, '
'250=0.04%\n'
' cpu : usr=17.70%, '
'sys=82.27%, ctx=18, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1536905,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=858MiB/s (899MB/s), '
'858MiB/s-858MiB/s '
'(899MB/s-899MB/s), io=6004MiB '
'(6295MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_249.output'),
'state': {'begin': 1666727088.0860672,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7894282341003418,
'end': 1666727088.8754954,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_249.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_249.state')},
'cmd_25': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_25.output'),
'state': {'begin': 1666726014.4914665,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.792306661605835,
'end': 1666726015.2837732,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_25.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_25.state')},
'cmd_250': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_250.output'),
'state': {'begin': 1666727090.9050484,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7911124229431152,
'end': 1666727091.6961608,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_250.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_250.state')},
'cmd_251': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_251.output'),
'state': {'begin': 1666727091.6984527,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.475045919418335,
'end': 1666727103.1734986,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_251.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_251.state')},
'cmd_252': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734192,\n'
' "timestamp_ms" : '
'1666734192584,\n'
' "time" : "Tue Oct 25 23:43:12 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6370635776,\n'
' "io_kbytes" : 6221324,\n'
' "bw_bytes" : 909960830,\n'
' "bw" : 888633,\n'
' "iops" : 222157.834595,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1555327,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1462,\n'
' "max" : 27873,\n'
' "mean" : 1640.823853,\n'
' "stddev" : '
'204.088642,\n'
' "N" : 1555328\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 240,\n'
' "max" : 140985,\n'
' "mean" : '
'16120.417763,\n'
' "stddev" : '
'5496.166937,\n'
' "N" : 1555330,\n'
' "percentile" : {\n'
' "1.000000" : 14144,\n'
' "5.000000" : 14400,\n'
' "10.000000" : '
'14528,\n'
' "20.000000" : '
'14528,\n'
' "30.000000" : '
'14656,\n'
' "40.000000" : '
'14656,\n'
' "50.000000" : '
'14656,\n'
' "60.000000" : '
'14784,\n'
' "70.000000" : '
'14784,\n'
' "80.000000" : '
'14912,\n'
' "90.000000" : '
'16512,\n'
' "95.000000" : '
'27008,\n'
' "99.000000" : '
'36096,\n'
' "99.500000" : '
'42752,\n'
' "99.900000" : '
'79360,\n'
' "99.950000" : '
'90624,\n'
' "99.990000" : '
'103936\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10810,\n'
' "max" : 142668,\n'
' "mean" : '
'17761.241593,\n'
' "stddev" : '
'5496.744254,\n'
' "N" : 1555330\n'
' },\n'
' "bw_min" : 887288,\n'
' "bw_max" : 890128,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'888840.714286,\n'
' "bw_dev" : 890.705375,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 221822,\n'
' "iops_max" : 222532,\n'
' "iops_mean" : '
'222210.285714,\n'
' "iops_stddev" : '
'222.724193,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 18.785714,\n'
' "sys_cpu" : 81.185714,\n'
' "ctx" : 13,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.012152,\n'
' "20" : 91.790730,\n'
' "50" : 8.009377,\n'
' "100" : 0.164274,\n'
' "250" : 0.023082,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=88037: Tue '
'Oct 25 23:43:12 2022\n'
' read: IOPS=222k, BW=868MiB/s '
'(910MB/s)(6076MiB/7001msec)\n'
' slat (nsec): min=1462, '
'max=27873, avg=1640.82, '
'stdev=204.09\n'
' clat (nsec): min=240, '
'max=140985, avg=16120.42, '
'stdev=5496.17\n'
' lat (usec): min=10, '
'max=142, avg=17.76, stdev= 5.50\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 15], '
'5.00th=[ 15], 10.00th=[ 15], '
'20.00th=[ 15],\n'
' | 30.00th=[ 15], '
'40.00th=[ 15], 50.00th=[ '
'15], 60.00th=[ 15],\n'
' | 70.00th=[ 15], '
'80.00th=[ 15], 90.00th=[ '
'17], 95.00th=[ 28],\n'
' | 99.00th=[ 37], '
'99.50th=[ 43], 99.90th=[ '
'80], 99.95th=[ 91],\n'
' | 99.99th=[ 104]\n'
' bw ( KiB/s): min=887288, '
'max=890128, per=100.00%, '
'avg=888840.71, stdev=890.71, '
'samples=14\n'
' iops : min=221822, '
'max=222532, avg=222210.29, '
'stdev=222.72, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=91.79%, 50=8.01%, '
'100=0.16%\n'
' lat (usec) : 250=0.02%\n'
' cpu : usr=18.79%, '
'sys=81.19%, ctx=13, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1555327,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=868MiB/s (910MB/s), '
'868MiB/s-868MiB/s '
'(910MB/s-910MB/s), io=6076MiB '
'(6371MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_252.output'),
'state': {'begin': 1666727103.1762161,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7970378398895264,
'end': 1666727103.973254,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_252.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_252.state')},
'cmd_253': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_253.output'),
'state': {'begin': 1666727104.959668,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.785696268081665,
'end': 1666727105.7453642,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_253.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_253.state')},
'cmd_254': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_254.output'),
'state': {'begin': 1666727105.752141,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring-xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.066343784332275,
'end': 1666727116.8184848,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_254.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_254.state')},
'cmd_255': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734206,\n'
' "timestamp_ms" : '
'1666734206301,\n'
' "time" : "Tue Oct 25 23:43:26 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'4069163008,\n'
' "io_kbytes" : 3973792,\n'
' "bw_bytes" : 581225968,\n'
' "bw" : 567603,\n'
' "iops" : 141900.442794,\n'
' "runtime" : 7001,\n'
' "total_ios" : 993445,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2274,\n'
' "max" : 1343589,\n'
' "mean" : 2510.766837,\n'
' "stddev" : '
'1415.574549,\n'
' "N" : 993445\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 240,\n'
' "max" : 1371542,\n'
' "mean" : '
'25199.312395,\n'
' "stddev" : '
'8781.217777,\n'
' "N" : 993448,\n'
' "percentile" : {\n'
' "1.000000" : 14016,\n'
' "5.000000" : 14272,\n'
' "10.000000" : '
'14400,\n'
' "20.000000" : '
'14656,\n'
' "30.000000" : '
'20864,\n'
' "40.000000" : '
'21376,\n'
' "50.000000" : '
'29568,\n'
' "60.000000" : '
'30080,\n'
' "70.000000" : '
'30080,\n'
' "80.000000" : '
'30336,\n'
' "90.000000" : '
'30592,\n'
' "95.000000" : '
'35072,\n'
' "99.000000" : '
'52992,\n'
' "99.500000" : '
'56576,\n'
' "99.900000" : '
'82432,\n'
' "99.950000" : '
'97792,\n'
' "99.990000" : '
'114176\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 14237,\n'
' "max" : 1373906,\n'
' "mean" : '
'27710.079092,\n'
' "stddev" : '
'8975.141547,\n'
' "N" : 993448\n'
' },\n'
' "bw_min" : 566520,\n'
' "bw_max" : 570888,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'567735.857143,\n'
' "bw_dev" : 1371.074472,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 141630,\n'
' "iops_max" : 142722,\n'
' "iops_mean" : '
'141933.928571,\n'
' "iops_stddev" : '
'342.789118,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 20.185714,\n'
' "sys_cpu" : 56.685714,\n'
' "ctx" : 253738,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 24.128261,\n'
' "50" : 74.194948,\n'
' "100" : 1.629783,\n'
' "250" : 0.044089,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=88076: Tue '
'Oct 25 23:43:26 2022\n'
' read: IOPS=142k, BW=554MiB/s '
'(581MB/s)(3881MiB/7001msec)\n'
' slat (usec): min=2, '
'max=1343, avg= 2.51, stdev= '
'1.42\n'
' clat (nsec): min=240, '
'max=1371.5k, avg=25199.31, '
'stdev=8781.22\n'
' lat (usec): min=14, '
'max=1373, avg=27.71, stdev= '
'8.98\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 15], '
'5.00th=[ 15], 10.00th=[ 15], '
'20.00th=[ 15],\n'
' | 30.00th=[ 21], '
'40.00th=[ 22], 50.00th=[ '
'30], 60.00th=[ 31],\n'
' | 70.00th=[ 31], '
'80.00th=[ 31], 90.00th=[ '
'31], 95.00th=[ 36],\n'
' | 99.00th=[ 53], '
'99.50th=[ 57], 99.90th=[ '
'83], 99.95th=[ 98],\n'
' | 99.99th=[ 115]\n'
' bw ( KiB/s): min=566520, '
'max=570888, per=100.00%, '
'avg=567735.86, stdev=1371.07, '
'samples=14\n'
' iops : min=141630, '
'max=142722, avg=141933.93, '
'stdev=342.79, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=24.13%, 50=74.19%, '
'100=1.63%\n'
' lat (usec) : 250=0.04%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=20.19%, '
'sys=56.69%, ctx=253738, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=993445,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=554MiB/s (581MB/s), '
'554MiB/s-554MiB/s '
'(581MB/s-581MB/s), io=3881MiB '
'(4069MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_255.output'),
'state': {'begin': 1666727116.8215501,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8406717777252197,
'end': 1666727117.662222,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_255.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_255.state')},
'cmd_256': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_256.output'),
'state': {'begin': 1666727118.686377,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7901842594146729,
'end': 1666727119.4765613,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_256.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_256.state')},
'cmd_257': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_257.output'),
'state': {'begin': 1666727119.4785686,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring-xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.056126117706299,
'end': 1666727130.5346947,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_257.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_257.state')},
'cmd_258': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734220,\n'
' "timestamp_ms" : '
'1666734220017,\n'
' "time" : "Tue Oct 25 23:43:40 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'4071936000,\n'
' "io_kbytes" : 3976500,\n'
' "bw_bytes" : 581622053,\n'
' "bw" : 567990,\n'
' "iops" : 141997.000429,\n'
' "runtime" : 7001,\n'
' "total_ios" : 994121,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2275,\n'
' "max" : 47789,\n'
' "mean" : 2511.122108,\n'
' "stddev" : '
'454.419258,\n'
' "N" : 994122\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 271,\n'
' "max" : 165681,\n'
' "mean" : '
'25181.048494,\n'
' "stddev" : '
'8477.785485,\n'
' "N" : 994124,\n'
' "percentile" : {\n'
' "1.000000" : 14016,\n'
' "5.000000" : 14144,\n'
' "10.000000" : '
'14272,\n'
' "20.000000" : '
'14656,\n'
' "30.000000" : '
'20864,\n'
' "40.000000" : '
'21376,\n'
' "50.000000" : '
'29568,\n'
' "60.000000" : '
'30080,\n'
' "70.000000" : '
'30080,\n'
' "80.000000" : '
'30336,\n'
' "90.000000" : '
'30592,\n'
' "95.000000" : '
'34560,\n'
' "99.000000" : '
'52992,\n'
' "99.500000" : '
'56576,\n'
' "99.900000" : '
'81408,\n'
' "99.950000" : '
'96768,\n'
' "99.990000" : '
'115200\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11572,\n'
' "max" : 168056,\n'
' "mean" : '
'27692.170811,\n'
' "stddev" : '
'8578.284311,\n'
' "N" : 994124\n'
' },\n'
' "bw_min" : 565416,\n'
' "bw_max" : 571520,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'568149.071429,\n'
' "bw_dev" : 1825.775977,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 141354,\n'
' "iops_max" : 142880,\n'
' "iops_mean" : '
'142037.214286,\n'
' "iops_stddev" : '
'456.460661,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 18.328571,\n'
' "sys_cpu" : 58.714286,\n'
' "ctx" : 253915,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 24.695384,\n'
' "50" : 73.607136,\n'
' "100" : 1.651710,\n'
' "250" : 0.043355,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=88116: Tue '
'Oct 25 23:43:40 2022\n'
' read: IOPS=142k, BW=555MiB/s '
'(582MB/s)(3883MiB/7001msec)\n'
' slat (nsec): min=2275, '
'max=47789, avg=2511.12, '
'stdev=454.42\n'
' clat (nsec): min=271, '
'max=165681, avg=25181.05, '
'stdev=8477.79\n'
' lat (usec): min=11, '
'max=168, avg=27.69, stdev= 8.58\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 15], '
'5.00th=[ 15], 10.00th=[ 15], '
'20.00th=[ 15],\n'
' | 30.00th=[ 21], '
'40.00th=[ 22], 50.00th=[ '
'30], 60.00th=[ 31],\n'
' | 70.00th=[ 31], '
'80.00th=[ 31], 90.00th=[ '
'31], 95.00th=[ 35],\n'
' | 99.00th=[ 53], '
'99.50th=[ 57], 99.90th=[ '
'82], 99.95th=[ 97],\n'
' | 99.99th=[ 116]\n'
' bw ( KiB/s): min=565416, '
'max=571520, per=100.00%, '
'avg=568149.07, stdev=1825.78, '
'samples=14\n'
' iops : min=141354, '
'max=142880, avg=142037.21, '
'stdev=456.46, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=24.70%, 50=73.61%, '
'100=1.65%\n'
' lat (usec) : 250=0.04%\n'
' cpu : usr=18.33%, '
'sys=58.71%, ctx=253915, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=994121,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=555MiB/s (582MB/s), '
'555MiB/s-555MiB/s '
'(582MB/s-582MB/s), io=3883MiB '
'(4072MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_258.output'),
'state': {'begin': 1666727130.5379033,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8043882846832275,
'end': 1666727131.3422916,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_258.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_258.state')},
'cmd_259': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_io_uring-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_259.output'),
'state': {'begin': 1666727132.333408,
'cmd': 'rm '
'/tmp/fio-output_4K_4_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7911653518676758,
'end': 1666727133.1245735,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_259.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_259.state')},
'cmd_26': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_26.output'),
'state': {'begin': 1666726015.2861967,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.375238418579102,
'end': 1666726026.6614351,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_26.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_26.state')},
'cmd_260': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_260.output'),
'state': {'begin': 1666727133.127158,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_io_uring-xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.059845924377441,
'end': 1666727144.1870039,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_260.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_260.state')},
'cmd_261': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734233,\n'
' "timestamp_ms" : '
'1666734233669,\n'
' "time" : "Tue Oct 25 23:43:53 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'4061618176,\n'
' "io_kbytes" : 3966424,\n'
' "bw_bytes" : 580148289,\n'
' "bw" : 566551,\n'
' "iops" : 141637.194686,\n'
' "runtime" : 7001,\n'
' "total_ios" : 991602,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2294,\n'
' "max" : 197260,\n'
' "mean" : 2527.106294,\n'
' "stddev" : '
'495.502687,\n'
' "N" : 991603\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 230,\n'
' "max" : 212398,\n'
' "mean" : '
'25205.674751,\n'
' "stddev" : '
'8552.840582,\n'
' "N" : 991605,\n'
' "percentile" : {\n'
' "1.000000" : 14016,\n'
' "5.000000" : 14144,\n'
' "10.000000" : '
'14272,\n'
' "20.000000" : '
'14528,\n'
' "30.000000" : '
'20864,\n'
' "40.000000" : '
'21120,\n'
' "50.000000" : '
'29824,\n'
' "60.000000" : '
'30080,\n'
' "70.000000" : '
'30336,\n'
' "80.000000" : '
'30336,\n'
' "90.000000" : '
'30592,\n'
' "95.000000" : '
'35072,\n'
' "99.000000" : '
'52480,\n'
' "99.500000" : '
'55552,\n'
' "99.900000" : '
'84480,\n'
' "99.950000" : '
'98816,\n'
' "99.990000" : '
'116224\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11661,\n'
' "max" : 215053,\n'
' "mean" : '
'27732.780705,\n'
' "stddev" : '
'8653.005043,\n'
' "N" : 991605\n'
' },\n'
' "bw_min" : 565176,\n'
' "bw_max" : 569450,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'566683.571429,\n'
' "bw_dev" : 1091.546871,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 141294,\n'
' "iops_max" : 142362,\n'
' "iops_mean" : '
'141670.857143,\n'
' "iops_stddev" : '
'272.789256,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 21.257143,\n'
' "sys_cpu" : 55.942857,\n'
' "ctx" : 253051,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 23.960924,\n'
' "50" : 74.303501,\n'
' "100" : 1.684345,\n'
' "250" : 0.046793,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=88156: Tue '
'Oct 25 23:43:53 2022\n'
' read: IOPS=142k, BW=553MiB/s '
'(580MB/s)(3873MiB/7001msec)\n'
' slat (nsec): min=2294, '
'max=197260, avg=2527.11, '
'stdev=495.50\n'
' clat (nsec): min=230, '
'max=212398, avg=25205.67, '
'stdev=8552.84\n'
' lat (usec): min=11, '
'max=215, avg=27.73, stdev= 8.65\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 15], '
'5.00th=[ 15], 10.00th=[ 15], '
'20.00th=[ 15],\n'
' | 30.00th=[ 21], '
'40.00th=[ 22], 50.00th=[ '
'30], 60.00th=[ 31],\n'
' | 70.00th=[ 31], '
'80.00th=[ 31], 90.00th=[ '
'31], 95.00th=[ 36],\n'
' | 99.00th=[ 53], '
'99.50th=[ 56], 99.90th=[ '
'85], 99.95th=[ 99],\n'
' | 99.99th=[ 117]\n'
' bw ( KiB/s): min=565176, '
'max=569450, per=100.00%, '
'avg=566683.57, stdev=1091.55, '
'samples=14\n'
' iops : min=141294, '
'max=142362, avg=141670.86, '
'stdev=272.79, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=23.96%, 50=74.30%, '
'100=1.68%\n'
' lat (usec) : 250=0.05%\n'
' cpu : usr=21.26%, '
'sys=55.94%, ctx=253051, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=991602,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=553MiB/s (580MB/s), '
'553MiB/s-553MiB/s '
'(580MB/s-580MB/s), io=3873MiB '
'(4062MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_261.output'),
'state': {'begin': 1666727144.19009,
'cmd': 'cat '
'/tmp/fio-output_4K_4_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7823362350463867,
'end': 1666727144.9724262,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_261.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_261.state')},
'cmd_262': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_libaio-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_262.output'),
'state': {'begin': 1666727145.9689424,
'cmd': 'rm '
'/tmp/fio-output_4K_4_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.784095048904419,
'end': 1666727146.7530375,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_262.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_262.state')},
'cmd_263': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_263.output'),
'state': {'begin': 1666727146.7549021,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_libaio-reference_0.txt"',
'cwd': 'None',
'elapsed': 11.070714235305786,
'end': 1666727157.8256164,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_263.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_263.state')},
'cmd_264': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=libaio, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734247,\n'
' "timestamp_ms" : '
'1666734247308,\n'
' "time" : "Tue Oct 25 23:44:07 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'5208461312,\n'
' "io_kbytes" : 5086388,\n'
' "bw_bytes" : 743959621,\n'
' "bw" : 726523,\n'
' "iops" : 181630.195686,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1271593,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1994,\n'
' "max" : 27552,\n'
' "mean" : 2273.459399,\n'
' "stddev" : '
'184.930489,\n'
' "N" : 1271594\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 6773,\n'
' "max" : 133701,\n'
' "mean" : '
'19356.888477,\n'
' "stddev" : '
'6100.168271,\n'
' "N" : 1271596,\n'
' "percentile" : {\n'
' "1.000000" : 14400,\n'
' "5.000000" : 14528,\n'
' "10.000000" : '
'14656,\n'
' "20.000000" : '
'14784,\n'
' "30.000000" : '
'15168,\n'
' "40.000000" : '
'15424,\n'
' "50.000000" : '
'18560,\n'
' "60.000000" : '
'21120,\n'
' "70.000000" : '
'21120,\n'
' "80.000000" : '
'21376,\n'
' "90.000000" : '
'21888,\n'
' "95.000000" : '
'30848,\n'
' "99.000000" : '
'42240,\n'
' "99.500000" : '
'47360,\n'
' "99.900000" : '
'77312,\n'
' "99.950000" : '
'91648,\n'
' "99.990000" : '
'104960\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11512,\n'
' "max" : 135855,\n'
' "mean" : '
'21630.347846,\n'
' "stddev" : '
'6127.066883,\n'
' "N" : 1271596\n'
' },\n'
' "bw_min" : 724216,\n'
' "bw_max" : 728976,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'726728.285714,\n'
' "bw_dev" : 1331.121877,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 181054,\n'
' "iops_max" : 182244,\n'
' "iops_mean" : '
'181682.071429,\n'
' "iops_stddev" : '
'332.780469,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 21.585714,\n'
' "sys_cpu" : 48.000000,\n'
' "ctx" : 330200,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 50.625082,\n'
' "50" : 49.125703,\n'
' "100" : 0.219253,\n'
' "250" : 0.023750,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 1806255,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 26402,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 26402,\n'
' "util" : 99.078934\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=88195: Tue '
'Oct 25 23:44:07 2022\n'
' read: IOPS=182k, BW=709MiB/s '
'(744MB/s)(4967MiB/7001msec)\n'
' slat (nsec): min=1994, '
'max=27552, avg=2273.46, '
'stdev=184.93\n'
' clat (usec): min=6, max=133, '
'avg=19.36, stdev= 6.10\n'
' lat (usec): min=11, '
'max=135, avg=21.63, stdev= 6.13\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 15], '
'5.00th=[ 15], 10.00th=[ 15], '
'20.00th=[ 15],\n'
' | 30.00th=[ 16], '
'40.00th=[ 16], 50.00th=[ '
'19], 60.00th=[ 22],\n'
' | 70.00th=[ 22], '
'80.00th=[ 22], 90.00th=[ '
'22], 95.00th=[ 31],\n'
' | 99.00th=[ 43], '
'99.50th=[ 48], 99.90th=[ '
'78], 99.95th=[ 92],\n'
' | 99.99th=[ 105]\n'
' bw ( KiB/s): min=724216, '
'max=728976, per=100.00%, '
'avg=726728.29, stdev=1331.12, '
'samples=14\n'
' iops : min=181054, '
'max=182244, avg=181682.07, '
'stdev=332.78, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=50.63%, 50=49.13%, 100=0.22%, '
'250=0.02%\n'
' cpu : usr=21.59%, '
'sys=48.00%, ctx=330200, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1271593,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=709MiB/s (744MB/s), '
'709MiB/s-709MiB/s '
'(744MB/s-744MB/s), io=4967MiB '
'(5208MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=1806255/0, '
'merge=0/0, ticks=26402/0, '
'in_queue=26402, util=99.08%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_264.output'),
'state': {'begin': 1666727157.8282857,
'cmd': 'cat '
'/tmp/fio-output_4K_4_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7942020893096924,
'end': 1666727158.6224878,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_264.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_264.state')},
'cmd_265': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_libaio-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_265.output'),
'state': {'begin': 1666727159.6133997,
'cmd': 'rm '
'/tmp/fio-output_4K_4_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7875704765319824,
'end': 1666727160.4009702,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_265.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_265.state')},
'cmd_266': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_266.output'),
'state': {'begin': 1666727160.403202,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_libaio-reference_1.txt"',
'cwd': 'None',
'elapsed': 11.080987930297852,
'end': 1666727171.48419,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_266.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_266.state')},
'cmd_267': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=libaio, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734260,\n'
' "timestamp_ms" : '
'1666734260967,\n'
' "time" : "Tue Oct 25 23:44:20 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'5129441280,\n'
' "io_kbytes" : 5009220,\n'
' "bw_bytes" : 732672658,\n'
' "bw" : 715500,\n'
' "iops" : 178874.589344,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1252301,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1342,\n'
' "max" : 38192,\n'
' "mean" : 2214.863789,\n'
' "stddev" : '
'1428.389632,\n'
' "N" : 1252302\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2564,\n'
' "max" : 191189,\n'
' "mean" : '
'19851.901731,\n'
' "stddev" : '
'5831.909239,\n'
' "N" : 1252304,\n'
' "percentile" : {\n'
' "1.000000" : 14016,\n'
' "5.000000" : 15040,\n'
' "10.000000" : '
'15552,\n'
' "20.000000" : '
'15808,\n'
' "30.000000" : '
'17280,\n'
' "40.000000" : '
'17536,\n'
' "50.000000" : '
'19072,\n'
' "60.000000" : '
'19584,\n'
' "70.000000" : '
'21376,\n'
' "80.000000" : '
'21376,\n'
' "90.000000" : '
'21888,\n'
' "95.000000" : '
'31616,\n'
' "99.000000" : '
'43264,\n'
' "99.500000" : '
'46336,\n'
' "99.900000" : '
'78336,\n'
' "99.950000" : '
'92672,\n'
' "99.990000" : '
'104960\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10289,\n'
' "max" : 195036,\n'
' "mean" : '
'22066.766055,\n'
' "stddev" : '
'5810.190816,\n'
' "N" : 1252304\n'
' },\n'
' "bw_min" : 705904,\n'
' "bw_max" : 720104,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'715783.785714,\n'
' "bw_dev" : 3968.759430,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 176476,\n'
' "iops_max" : 180026,\n'
' "iops_mean" : '
'178945.857143,\n'
' "iops_stddev" : '
'992.094820,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.257143,\n'
' "sys_cpu" : 45.128571,\n'
' "ctx" : 319706,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.063084,\n'
' "20" : 61.387717,\n'
' "50" : 38.299898,\n'
' "100" : 0.221991,\n'
' "250" : 0.026990,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 1786396,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 28771,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 28771,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=88233: Tue '
'Oct 25 23:44:20 2022\n'
' read: IOPS=179k, BW=699MiB/s '
'(733MB/s)(4892MiB/7001msec)\n'
' slat (nsec): min=1342, '
'max=38192, avg=2214.86, '
'stdev=1428.39\n'
' clat (usec): min=2, max=191, '
'avg=19.85, stdev= 5.83\n'
' lat (usec): min=10, '
'max=195, avg=22.07, stdev= 5.81\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 15], '
'5.00th=[ 16], 10.00th=[ 16], '
'20.00th=[ 16],\n'
' | 30.00th=[ 18], '
'40.00th=[ 18], 50.00th=[ '
'20], 60.00th=[ 20],\n'
' | 70.00th=[ 22], '
'80.00th=[ 22], 90.00th=[ '
'22], 95.00th=[ 32],\n'
' | 99.00th=[ 44], '
'99.50th=[ 47], 99.90th=[ '
'79], 99.95th=[ 93],\n'
' | 99.99th=[ 105]\n'
' bw ( KiB/s): min=705904, '
'max=720104, per=100.00%, '
'avg=715783.79, stdev=3968.76, '
'samples=14\n'
' iops : min=176476, '
'max=180026, avg=178945.86, '
'stdev=992.09, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.06%, 20=61.39%, 50=38.30%, '
'100=0.22%\n'
' lat (usec) : 250=0.03%\n'
' cpu : usr=17.26%, '
'sys=45.13%, ctx=319706, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1252301,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=699MiB/s (733MB/s), '
'699MiB/s-699MiB/s '
'(733MB/s-733MB/s), io=4892MiB '
'(5129MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=1786396/0, '
'merge=0/0, ticks=28771/0, '
'in_queue=28771, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_267.output'),
'state': {'begin': 1666727171.4907925,
'cmd': 'cat '
'/tmp/fio-output_4K_4_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7877211570739746,
'end': 1666727172.2785137,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_267.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_267.state')},
'cmd_268': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_libaio-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_268.output'),
'state': {'begin': 1666727173.265863,
'cmd': 'rm '
'/tmp/fio-output_4K_4_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7897942066192627,
'end': 1666727174.0556571,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_268.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_268.state')},
'cmd_269': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_269.output'),
'state': {'begin': 1666727174.0578525,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_libaio-reference_2.txt"',
'cwd': 'None',
'elapsed': 11.066452026367188,
'end': 1666727185.1243045,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_269.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_269.state')},
'cmd_27': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733116,\n'
' "timestamp_ms" : '
'1666733116091,\n'
' "time" : "Tue Oct 25 23:25:16 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2407424000,\n'
' "io_kbytes" : 2351000,\n'
' "bw_bytes" : 343868590,\n'
' "bw" : 335809,\n'
' "iops" : 83952.292530,\n'
' "runtime" : 7001,\n'
' "total_ios" : 587750,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 711,\n'
' "max" : 11492,\n'
' "mean" : 925.037659,\n'
' "stddev" : 79.223115,\n'
' "N" : 587750\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 3837,\n'
' "max" : 176060,\n'
' "mean" : 10698.287205,\n'
' "stddev" : '
'2205.678474,\n'
' "N" : 587750,\n'
' "percentile" : {\n'
' "1.000000" : 9920,\n'
' "5.000000" : 10048,\n'
' "10.000000" : 10048,\n'
' "20.000000" : 10176,\n'
' "30.000000" : 10304,\n'
' "40.000000" : 10304,\n'
' "50.000000" : 10432,\n'
' "60.000000" : 10432,\n'
' "70.000000" : 10560,\n'
' "80.000000" : 10560,\n'
' "90.000000" : 10688,\n'
' "95.000000" : 11072,\n'
' "99.000000" : 19840,\n'
' "99.500000" : 27520,\n'
' "99.900000" : 28032,\n'
' "99.950000" : 43776,\n'
' "99.990000" : 67072\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10570,\n'
' "max" : 179687,\n'
' "mean" : 11623.324864,\n'
' "stddev" : '
'2208.087793,\n'
' "N" : 587750\n'
' },\n'
' "bw_min" : 335496,\n'
' "bw_max" : 336312,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'335897.642857,\n'
' "bw_dev" : 239.573213,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 83874,\n'
' "iops_max" : 84078,\n'
' "iops_mean" : '
'83974.357143,\n'
' "iops_stddev" : '
'59.925091,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 20,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 3.303956,\n'
' "20" : 95.712123,\n'
' "50" : 0.938494,\n'
' "100" : 0.044747,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=84733: Tue Oct 25 23:25:16 '
'2022\n'
' read: IOPS=84.0k, BW=328MiB/s '
'(344MB/s)(2296MiB/7001msec)\n'
' slat (nsec): min=711, '
'max=11492, avg=925.04, '
'stdev=79.22\n'
' clat (usec): min=3, max=176, '
'avg=10.70, stdev= 2.21\n'
' lat (usec): min=10, max=179, '
'avg=11.62, stdev= 2.21\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9920], '
'5.00th=[10048], 10.00th=[10048], '
'20.00th=[10176],\n'
' | 30.00th=[10304], '
'40.00th=[10304], 50.00th=[10432], '
'60.00th=[10432],\n'
' | 70.00th=[10560], '
'80.00th=[10560], 90.00th=[10688], '
'95.00th=[11072],\n'
' | 99.00th=[19840], '
'99.50th=[27520], 99.90th=[28032], '
'99.95th=[43776],\n'
' | 99.99th=[67072]\n'
' bw ( KiB/s): min=335496, '
'max=336312, per=100.00%, '
'avg=335897.64, stdev=239.57, '
'samples=14\n'
' iops : min=83874, '
'max=84078, avg=83974.36, '
'stdev=59.93, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=3.30%, 20=95.71%, 50=0.94%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=20, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=587750,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=328MiB/s (344MB/s), '
'328MiB/s-328MiB/s '
'(344MB/s-344MB/s), io=2296MiB '
'(2407MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_27.output'),
'state': {'begin': 1666726026.6640089,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7969875335693359,
'end': 1666726027.4609964,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_27.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_27.state')},
'cmd_270': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=libaio, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734274,\n'
' "timestamp_ms" : '
'1666734274606,\n'
' "time" : "Tue Oct 25 23:44:34 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'5094621184,\n'
' "io_kbytes" : 4975216,\n'
' "bw_bytes" : 727699069,\n'
' "bw" : 710643,\n'
' "iops" : 177660.477075,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1243801,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1322,\n'
' "max" : 179357,\n'
' "mean" : 2270.165353,\n'
' "stddev" : '
'1482.663537,\n'
' "N" : 1243801\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2415,\n'
' "max" : 1314775,\n'
' "mean" : '
'19921.951998,\n'
' "stddev" : '
'6421.548737,\n'
' "N" : 1243804,\n'
' "percentile" : {\n'
' "1.000000" : 14016,\n'
' "5.000000" : 15040,\n'
' "10.000000" : '
'15552,\n'
' "20.000000" : '
'15808,\n'
' "30.000000" : '
'17280,\n'
' "40.000000" : '
'17536,\n'
' "50.000000" : '
'19072,\n'
' "60.000000" : '
'19328,\n'
' "70.000000" : '
'21120,\n'
' "80.000000" : '
'21376,\n'
' "90.000000" : '
'23680,\n'
' "95.000000" : '
'31872,\n'
' "99.000000" : '
'43264,\n'
' "99.500000" : '
'47360,\n'
' "99.900000" : '
'79360,\n'
' "99.950000" : '
'93696,\n'
' "99.990000" : '
'104960\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9989,\n'
' "max" : 1316138,\n'
' "mean" : '
'22192.118014,\n'
' "stddev" : '
'6367.348917,\n'
' "N" : 1243804\n'
' },\n'
' "bw_min" : 653496,\n'
' "bw_max" : 722680,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'710909.000000,\n'
' "bw_dev" : '
'18576.458195,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 163374,\n'
' "iops_max" : 180670,\n'
' "iops_mean" : '
'177727.285714,\n'
' "iops_stddev" : '
'4644.186298,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.771429,\n'
' "sys_cpu" : 45.885714,\n'
' "ctx" : 313922,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.049365,\n'
' "20" : 62.858367,\n'
' "50" : 36.778311,\n'
' "100" : 0.285978,\n'
' "250" : 0.027336,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 1774792,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 28485,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 28485,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=88271: Tue '
'Oct 25 23:44:34 2022\n'
' read: IOPS=178k, BW=694MiB/s '
'(728MB/s)(4859MiB/7001msec)\n'
' slat (nsec): min=1322, '
'max=179357, avg=2270.17, '
'stdev=1482.66\n'
' clat (usec): min=2, '
'max=1314, avg=19.92, stdev= '
'6.42\n'
' lat (usec): min=9, '
'max=1316, avg=22.19, stdev= '
'6.37\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 15], '
'5.00th=[ 16], 10.00th=[ 16], '
'20.00th=[ 16],\n'
' | 30.00th=[ 18], '
'40.00th=[ 18], 50.00th=[ '
'20], 60.00th=[ 20],\n'
' | 70.00th=[ 22], '
'80.00th=[ 22], 90.00th=[ '
'24], 95.00th=[ 32],\n'
' | 99.00th=[ 44], '
'99.50th=[ 48], 99.90th=[ '
'80], 99.95th=[ 94],\n'
' | 99.99th=[ 105]\n'
' bw ( KiB/s): min=653496, '
'max=722680, per=100.00%, '
'avg=710909.00, stdev=18576.46, '
'samples=14\n'
' iops : min=163374, '
'max=180670, avg=177727.29, '
'stdev=4644.19, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.05%, 20=62.86%, 50=36.78%, '
'100=0.29%\n'
' lat (usec) : 250=0.03%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=17.77%, '
'sys=45.89%, ctx=313922, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1243801,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=694MiB/s (728MB/s), '
'694MiB/s-694MiB/s '
'(728MB/s-728MB/s), io=4859MiB '
'(5095MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=1774792/0, '
'merge=0/0, ticks=28485/0, '
'in_queue=28485, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_270.output'),
'state': {'begin': 1666727185.1269662,
'cmd': 'cat '
'/tmp/fio-output_4K_4_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.8012464046478271,
'end': 1666727185.9282126,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_270.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_270.state')},
'cmd_271': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_libaio-bdev_aio_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_271.output'),
'state': {'begin': 1666727187.9529927,
'cmd': 'rm '
'/tmp/fio-output_4K_4_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.787787675857544,
'end': 1666727188.7407804,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_271.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_271.state')},
'cmd_272': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_272.output'),
'state': {'begin': 1666727188.742865,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_libaio-bdev_aio_0.txt"',
'cwd': 'None',
'elapsed': 11.424794912338257,
'end': 1666727200.16766,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_272.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_272.state')},
'cmd_273': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734289,\n'
' "timestamp_ms" : '
'1666734289535,\n'
' "time" : "Tue Oct 25 23:44:49 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6368309248,\n'
' "io_kbytes" : 6219052,\n'
' "bw_bytes" : 909628517,\n'
' "bw" : 888309,\n'
' "iops" : 222076.846165,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1554760,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1523,\n'
' "max" : 20198,\n'
' "mean" : 1636.536629,\n'
' "stddev" : '
'223.818184,\n'
' "N" : 1554760\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 251,\n'
' "max" : 148058,\n'
' "mean" : '
'16084.288581,\n'
' "stddev" : '
'5476.140778,\n'
' "N" : 1554763,\n'
' "percentile" : {\n'
' "1.000000" : 13504,\n'
' "5.000000" : 14400,\n'
' "10.000000" : '
'14528,\n'
' "20.000000" : '
'14528,\n'
' "30.000000" : '
'14656,\n'
' "40.000000" : '
'14656,\n'
' "50.000000" : '
'14656,\n'
' "60.000000" : '
'14784,\n'
' "70.000000" : '
'14784,\n'
' "80.000000" : '
'14912,\n'
' "90.000000" : '
'16512,\n'
' "95.000000" : '
'27008,\n'
' "99.000000" : '
'36096,\n'
' "99.500000" : '
'42752,\n'
' "99.900000" : '
'79360,\n'
' "99.950000" : '
'90624,\n'
' "99.990000" : '
'104960\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9618,\n'
' "max" : 149671,\n'
' "mean" : '
'17720.825126,\n'
' "stddev" : '
'5474.918573,\n'
' "N" : 1554763\n'
' },\n'
' "bw_min" : 886712,\n'
' "bw_max" : 890216,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'888525.785714,\n'
' "bw_dev" : 1043.348321,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 221678,\n'
' "iops_max" : 222554,\n'
' "iops_mean" : '
'222131.428571,\n'
' "iops_stddev" : '
'260.832311,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 73.700000,\n'
' "sys_cpu" : 26.285714,\n'
' "ctx" : 4,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.372984,\n'
' "2" : 0.960470,\n'
' "4" : 98.666547,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.063933,\n'
' "20" : 92.064048,\n'
' "50" : 7.686588,\n'
' "100" : 0.157516,\n'
' "250" : 0.025663,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=88319: Tue '
'Oct 25 23:44:49 2022\n'
' read: IOPS=222k, BW=867MiB/s '
'(910MB/s)(6073MiB/7001msec)\n'
' slat (nsec): min=1523, '
'max=20198, avg=1636.54, '
'stdev=223.82\n'
' clat (nsec): min=251, '
'max=148058, avg=16084.29, '
'stdev=5476.14\n'
' lat (usec): min=9, max=149, '
'avg=17.72, stdev= 5.47\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 14], '
'5.00th=[ 15], 10.00th=[ 15], '
'20.00th=[ 15],\n'
' | 30.00th=[ 15], '
'40.00th=[ 15], 50.00th=[ '
'15], 60.00th=[ 15],\n'
' | 70.00th=[ 15], '
'80.00th=[ 15], 90.00th=[ '
'17], 95.00th=[ 28],\n'
' | 99.00th=[ 37], '
'99.50th=[ 43], 99.90th=[ '
'80], 99.95th=[ 91],\n'
' | 99.99th=[ 105]\n'
' bw ( KiB/s): min=886712, '
'max=890216, per=100.00%, '
'avg=888525.79, stdev=1043.35, '
'samples=14\n'
' iops : min=221678, '
'max=222554, avg=222131.43, '
'stdev=260.83, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.06%, 20=92.06%, 50=7.69%, '
'100=0.16%\n'
' lat (usec) : 250=0.03%\n'
' cpu : usr=73.70%, '
'sys=26.29%, ctx=4, majf=0, '
'minf=0\n'
' IO depths : 1=0.4%, 2=1.0%, '
'4=98.7%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1554760,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=867MiB/s (910MB/s), '
'867MiB/s-867MiB/s '
'(910MB/s-910MB/s), io=6073MiB '
'(6368MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_273.output'),
'state': {'begin': 1666727200.1704807,
'cmd': 'cat '
'/tmp/fio-output_4K_4_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.7871673107147217,
'end': 1666727200.957648,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_273.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_273.state')},
'cmd_274': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_libaio-bdev_aio_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_274.output'),
'state': {'begin': 1666727202.9961932,
'cmd': 'rm '
'/tmp/fio-output_4K_4_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.801347017288208,
'end': 1666727203.7975402,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_274.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_274.state')},
'cmd_275': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_275.output'),
'state': {'begin': 1666727203.80041,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_libaio-bdev_aio_1.txt"',
'cwd': 'None',
'elapsed': 11.53487515449524,
'end': 1666727215.3352852,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_275.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_275.state')},
'cmd_276': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734304,\n'
' "timestamp_ms" : '
'1666734304702,\n'
' "time" : "Tue Oct 25 23:45:04 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6271488000,\n'
' "io_kbytes" : 6124500,\n'
' "bw_bytes" : 895798885,\n'
' "bw" : 874803,\n'
' "iops" : 218700.328524,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1531121,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1643,\n'
' "max" : 62267,\n'
' "mean" : 1883.836522,\n'
' "stddev" : '
'167.538851,\n'
' "N" : 1531122\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 290,\n'
' "max" : 141535,\n'
' "mean" : '
'16162.363481,\n'
' "stddev" : '
'6497.223878,\n'
' "N" : 1531121,\n'
' "percentile" : {\n'
' "1.000000" : 10432,\n'
' "5.000000" : 10432,\n'
' "10.000000" : '
'10560,\n'
' "20.000000" : '
'12480,\n'
' "30.000000" : '
'12864,\n'
' "40.000000" : '
'14272,\n'
' "50.000000" : '
'14528,\n'
' "60.000000" : '
'14784,\n'
' "70.000000" : '
'17024,\n'
' "80.000000" : '
'19328,\n'
' "90.000000" : '
'21376,\n'
' "95.000000" : '
'28800,\n'
' "99.000000" : '
'41216,\n'
' "99.500000" : '
'44800,\n'
' "99.900000" : '
'77312,\n'
' "99.950000" : '
'94720,\n'
' "99.990000" : '
'121344\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9808,\n'
' "max" : 143308,\n'
' "mean" : '
'18046.199945,\n'
' "stddev" : '
'6499.210762,\n'
' "N" : 1531121\n'
' },\n'
' "bw_min" : 873272,\n'
' "bw_max" : 877680,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'875229.571429,\n'
' "bw_dev" : 1466.632654,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 218318,\n'
' "iops_max" : 219420,\n'
' "iops_mean" : '
'218807.285714,\n'
' "iops_stddev" : '
'366.406758,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 77.814286,\n'
' "sys_cpu" : 22.157143,\n'
' "ctx" : 22,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.441180,\n'
' "4" : 99.548239,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.063418,\n'
' "20" : 80.184910,\n'
' "50" : 19.526347,\n'
' "100" : 0.182611,\n'
' "250" : 0.041016,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=88368: Tue '
'Oct 25 23:45:04 2022\n'
' read: IOPS=219k, BW=854MiB/s '
'(896MB/s)(5981MiB/7001msec)\n'
' slat (nsec): min=1643, '
'max=62267, avg=1883.84, '
'stdev=167.54\n'
' clat (nsec): min=290, '
'max=141535, avg=16162.36, '
'stdev=6497.22\n'
' lat (usec): min=9, max=143, '
'avg=18.05, stdev= 6.50\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 11], '
'5.00th=[ 11], 10.00th=[ 11], '
'20.00th=[ 13],\n'
' | 30.00th=[ 13], '
'40.00th=[ 15], 50.00th=[ '
'15], 60.00th=[ 15],\n'
' | 70.00th=[ 18], '
'80.00th=[ 20], 90.00th=[ '
'22], 95.00th=[ 29],\n'
' | 99.00th=[ 42], '
'99.50th=[ 45], 99.90th=[ '
'78], 99.95th=[ 95],\n'
' | 99.99th=[ 122]\n'
' bw ( KiB/s): min=873272, '
'max=877680, per=100.00%, '
'avg=875229.57, stdev=1466.63, '
'samples=14\n'
' iops : min=218318, '
'max=219420, avg=218807.29, '
'stdev=366.41, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 10=0.06%, '
'20=80.18%, 50=19.53%, 100=0.18%, '
'250=0.04%\n'
' cpu : usr=77.81%, '
'sys=22.16%, ctx=22, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.4%, '
'4=99.5%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1531121,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=854MiB/s (896MB/s), '
'854MiB/s-854MiB/s '
'(896MB/s-896MB/s), io=5981MiB '
'(6271MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_276.output'),
'state': {'begin': 1666727215.3379507,
'cmd': 'cat '
'/tmp/fio-output_4K_4_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7949707508087158,
'end': 1666727216.1329215,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_276.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_276.state')},
'cmd_277': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_libaio-bdev_aio_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_277.output'),
'state': {'begin': 1666727218.1851926,
'cmd': 'rm '
'/tmp/fio-output_4K_4_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.7935588359832764,
'end': 1666727218.9787514,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_277.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_277.state')},
'cmd_278': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_278.output'),
'state': {'begin': 1666727218.981152,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_libaio-bdev_aio_2.txt"',
'cwd': 'None',
'elapsed': 11.622839212417603,
'end': 1666727230.6039913,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_278.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_278.state')},
'cmd_279': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734319,\n'
' "timestamp_ms" : '
'1666734319976,\n'
' "time" : "Tue Oct 25 23:45:19 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6364598272,\n'
' "io_kbytes" : 6215428,\n'
' "bw_bytes" : 909098453,\n'
' "bw" : 887791,\n'
' "iops" : 221947.293244,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1553853,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1513,\n'
' "max" : 25969,\n'
' "mean" : 1631.607934,\n'
' "stddev" : '
'243.107289,\n'
' "N" : 1553854\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 261,\n'
' "max" : 131006,\n'
' "mean" : '
'16099.061237,\n'
' "stddev" : '
'5450.853875,\n'
' "N" : 1553853,\n'
' "percentile" : {\n'
' "1.000000" : 13504,\n'
' "5.000000" : 14400,\n'
' "10.000000" : '
'14528,\n'
' "20.000000" : '
'14528,\n'
' "30.000000" : '
'14656,\n'
' "40.000000" : '
'14656,\n'
' "50.000000" : '
'14656,\n'
' "60.000000" : '
'14784,\n'
' "70.000000" : '
'14784,\n'
' "80.000000" : '
'14912,\n'
' "90.000000" : '
'16512,\n'
' "95.000000" : '
'27008,\n'
' "99.000000" : '
'36096,\n'
' "99.500000" : '
'42752,\n'
' "99.900000" : '
'79360,\n'
' "99.950000" : '
'88576,\n'
' "99.990000" : '
'103936\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9708,\n'
' "max" : 132579,\n'
' "mean" : '
'17730.669197,\n'
' "stddev" : '
'5450.382413,\n'
' "N" : 1553853\n'
' },\n'
' "bw_min" : 886560,\n'
' "bw_max" : 889792,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'888029.857143,\n'
' "bw_dev" : 1001.853195,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 221640,\n'
' "iops_max" : 222448,\n'
' "iops_mean" : '
'222007.428571,\n'
' "iops_stddev" : '
'250.424168,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 69.928571,\n'
' "sys_cpu" : 30.042857,\n'
' "ctx" : 4,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.372300,\n'
' "2" : 0.969268,\n'
' "4" : 98.658432,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.058757,\n'
' "20" : 91.962818,\n'
' "50" : 7.797906,\n'
' "100" : 0.153618,\n'
' "250" : 0.024327,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=88418: Tue '
'Oct 25 23:45:19 2022\n'
' read: IOPS=222k, BW=867MiB/s '
'(909MB/s)(6070MiB/7001msec)\n'
' slat (nsec): min=1513, '
'max=25969, avg=1631.61, '
'stdev=243.11\n'
' clat (nsec): min=261, '
'max=131006, avg=16099.06, '
'stdev=5450.85\n'
' lat (usec): min=9, max=132, '
'avg=17.73, stdev= 5.45\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 14], '
'5.00th=[ 15], 10.00th=[ 15], '
'20.00th=[ 15],\n'
' | 30.00th=[ 15], '
'40.00th=[ 15], 50.00th=[ '
'15], 60.00th=[ 15],\n'
' | 70.00th=[ 15], '
'80.00th=[ 15], 90.00th=[ '
'17], 95.00th=[ 28],\n'
' | 99.00th=[ 37], '
'99.50th=[ 43], 99.90th=[ '
'80], 99.95th=[ 89],\n'
' | 99.99th=[ 104]\n'
' bw ( KiB/s): min=886560, '
'max=889792, per=100.00%, '
'avg=888029.86, stdev=1001.85, '
'samples=14\n'
' iops : min=221640, '
'max=222448, avg=222007.43, '
'stdev=250.42, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 10=0.06%, '
'20=91.96%, 50=7.80%, 100=0.15%, '
'250=0.02%\n'
' cpu : usr=69.93%, '
'sys=30.04%, ctx=4, majf=0, '
'minf=0\n'
' IO depths : 1=0.4%, 2=1.0%, '
'4=98.7%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1553853,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=867MiB/s (909MB/s), '
'867MiB/s-867MiB/s '
'(909MB/s-909MB/s), io=6070MiB '
'(6365MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_279.output'),
'state': {'begin': 1666727230.6067412,
'cmd': 'cat '
'/tmp/fio-output_4K_4_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.794581413269043,
'end': 1666727231.4013226,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_279.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_279.state')},
'cmd_28': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_28.output'),
'state': {'begin': 1666726028.4532728,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7852113246917725,
'end': 1666726029.2384841,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_28.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_28.state')},
'cmd_280': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_280.output'),
'state': {'begin': 1666727233.4648118,
'cmd': 'rm '
'/tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7895946502685547,
'end': 1666727234.2544065,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_280.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_280.state')},
'cmd_281': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_281.output'),
'state': {'begin': 1666727234.256411,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.527220249176025,
'end': 1666727245.7836313,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_281.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_281.state')},
'cmd_282': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734335,\n'
' "timestamp_ms" : '
'1666734335155,\n'
' "time" : "Tue Oct 25 23:45:35 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6271553536,\n'
' "io_kbytes" : 6124564,\n'
' "bw_bytes" : 895808246,\n'
' "bw" : 874812,\n'
' "iops" : 218702.756749,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1531138,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1552,\n'
' "max" : 172423,\n'
' "mean" : 2025.800648,\n'
' "stddev" : '
'791.208125,\n'
' "N" : 1531138\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 651,\n'
' "max" : 183304,\n'
' "mean" : '
'15858.742410,\n'
' "stddev" : '
'5803.263051,\n'
' "N" : 1531141,\n'
' "percentile" : {\n'
' "1.000000" : 10176,\n'
' "5.000000" : 10560,\n'
' "10.000000" : '
'12992,\n'
' "20.000000" : '
'13760,\n'
' "30.000000" : '
'14016,\n'
' "40.000000" : '
'14272,\n'
' "50.000000" : '
'14528,\n'
' "60.000000" : '
'14784,\n'
' "70.000000" : '
'15168,\n'
' "80.000000" : '
'16192,\n'
' "90.000000" : '
'17792,\n'
' "95.000000" : '
'27520,\n'
' "99.000000" : '
'38656,\n'
' "99.500000" : '
'42752,\n'
' "99.900000" : '
'79360,\n'
' "99.950000" : '
'90624,\n'
' "99.990000" : '
'103936\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9658,\n'
' "max" : 186690,\n'
' "mean" : '
'17884.542243,\n'
' "stddev" : '
'5775.342587,\n'
' "N" : 1531141\n'
' },\n'
' "bw_min" : 872848,\n'
' "bw_max" : 876888,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'875033.785714,\n'
' "bw_dev" : 1002.540326,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 218212,\n'
' "iops_max" : 219222,\n'
' "iops_mean" : '
'218758.428571,\n'
' "iops_stddev" : '
'250.748024,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 42.671429,\n'
' "sys_cpu" : 57.300000,\n'
' "ctx" : 12,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.273522,\n'
' "2" : 6.079334,\n'
' "4" : 93.647143,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.020834,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.145970,\n'
' "20" : 92.134086,\n'
' "50" : 7.513366,\n'
' "100" : 0.155897,\n'
' "250" : 0.024753,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=88468: Tue '
'Oct 25 23:45:35 2022\n'
' read: IOPS=219k, BW=854MiB/s '
'(896MB/s)(5981MiB/7001msec)\n'
' slat (nsec): min=1552, '
'max=172423, avg=2025.80, '
'stdev=791.21\n'
' clat (nsec): min=651, '
'max=183304, avg=15858.74, '
'stdev=5803.26\n'
' lat (usec): min=9, max=186, '
'avg=17.88, stdev= 5.78\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 11], '
'5.00th=[ 11], 10.00th=[ 13], '
'20.00th=[ 14],\n'
' | 30.00th=[ 15], '
'40.00th=[ 15], 50.00th=[ '
'15], 60.00th=[ 15],\n'
' | 70.00th=[ 16], '
'80.00th=[ 17], 90.00th=[ '
'18], 95.00th=[ 28],\n'
' | 99.00th=[ 39], '
'99.50th=[ 43], 99.90th=[ '
'80], 99.95th=[ 91],\n'
' | 99.99th=[ 104]\n'
' bw ( KiB/s): min=872848, '
'max=876888, per=100.00%, '
'avg=875033.79, stdev=1002.54, '
'samples=14\n'
' iops : min=218212, '
'max=219222, avg=218758.43, '
'stdev=250.75, samples=14\n'
' lat (nsec) : 750=0.02%, '
'1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.15%, 20=92.13%, 50=7.51%, '
'100=0.16%\n'
' lat (usec) : 250=0.02%\n'
' cpu : usr=42.67%, '
'sys=57.30%, ctx=12, majf=0, '
'minf=0\n'
' IO depths : 1=0.3%, 2=6.1%, '
'4=93.6%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1531138,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=854MiB/s (896MB/s), '
'854MiB/s-854MiB/s '
'(896MB/s-896MB/s), io=5981MiB '
'(6272MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_282.output'),
'state': {'begin': 1666727245.7862487,
'cmd': 'cat '
'/tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.789813756942749,
'end': 1666727246.5760624,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_282.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_282.state')},
'cmd_283': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_283.output'),
'state': {'begin': 1666727248.5907779,
'cmd': 'rm '
'/tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7839603424072266,
'end': 1666727249.3747382,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_283.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_283.state')},
'cmd_284': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_284.output'),
'state': {'begin': 1666727249.3769617,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.562498569488525,
'end': 1666727260.9394603,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_284.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_284.state')},
'cmd_285': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734350,\n'
' "timestamp_ms" : '
'1666734350304,\n'
' "time" : "Tue Oct 25 23:45:50 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6267457536,\n'
' "io_kbytes" : 6120564,\n'
' "bw_bytes" : 895223187,\n'
' "bw" : 874241,\n'
' "iops" : 218559.777175,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1530137,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1532,\n'
' "max" : 20007,\n'
' "mean" : 2014.375794,\n'
' "stddev" : '
'757.866366,\n'
' "N" : 1530138\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 641,\n'
' "max" : 144321,\n'
' "mean" : '
'15884.766559,\n'
' "stddev" : '
'5830.698008,\n'
' "N" : 1530137,\n'
' "percentile" : {\n'
' "1.000000" : 10176,\n'
' "5.000000" : 10560,\n'
' "10.000000" : '
'12992,\n'
' "20.000000" : '
'13760,\n'
' "30.000000" : '
'14016,\n'
' "40.000000" : '
'14272,\n'
' "50.000000" : '
'14528,\n'
' "60.000000" : '
'14784,\n'
' "70.000000" : '
'15168,\n'
' "80.000000" : '
'16192,\n'
' "90.000000" : '
'18048,\n'
' "95.000000" : '
'27520,\n'
' "99.000000" : '
'38656,\n'
' "99.500000" : '
'42752,\n'
' "99.900000" : '
'80384,\n'
' "99.950000" : '
'90624,\n'
' "99.990000" : '
'103936\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9969,\n'
' "max" : 145914,\n'
' "mean" : '
'17899.142622,\n'
' "stddev" : '
'5795.903965,\n'
' "N" : 1530137\n'
' },\n'
' "bw_min" : 871464,\n'
' "bw_max" : 876000,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'874451.071429,\n'
' "bw_dev" : 1455.933772,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 217866,\n'
' "iops_max" : 219000,\n'
' "iops_mean" : '
'218612.857143,\n'
' "iops_stddev" : '
'364.172161,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 41.014286,\n'
' "sys_cpu" : 58.957143,\n'
' "ctx" : 6,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.280106,\n'
' "2" : 6.138274,\n'
' "4" : 93.581620,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.018953,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.173122,\n'
' "20" : 92.036465,\n'
' "50" : 7.573636,\n'
' "100" : 0.166521,\n'
' "250" : 0.026403,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=88518: Tue '
'Oct 25 23:45:50 2022\n'
' read: IOPS=219k, BW=854MiB/s '
'(895MB/s)(5977MiB/7001msec)\n'
' slat (nsec): min=1532, '
'max=20007, avg=2014.38, '
'stdev=757.87\n'
' clat (nsec): min=641, '
'max=144321, avg=15884.77, '
'stdev=5830.70\n'
' lat (usec): min=9, max=145, '
'avg=17.90, stdev= 5.80\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 11], '
'5.00th=[ 11], 10.00th=[ 13], '
'20.00th=[ 14],\n'
' | 30.00th=[ 15], '
'40.00th=[ 15], 50.00th=[ '
'15], 60.00th=[ 15],\n'
' | 70.00th=[ 16], '
'80.00th=[ 17], 90.00th=[ '
'19], 95.00th=[ 28],\n'
' | 99.00th=[ 39], '
'99.50th=[ 43], 99.90th=[ '
'81], 99.95th=[ 91],\n'
' | 99.99th=[ 104]\n'
' bw ( KiB/s): min=871464, '
'max=876000, per=100.00%, '
'avg=874451.07, stdev=1455.93, '
'samples=14\n'
' iops : min=217866, '
'max=219000, avg=218612.86, '
'stdev=364.17, samples=14\n'
' lat (nsec) : 750=0.02%, '
'1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.17%, 20=92.04%, 50=7.57%, '
'100=0.17%\n'
' lat (usec) : 250=0.03%\n'
' cpu : usr=41.01%, '
'sys=58.96%, ctx=6, majf=0, '
'minf=0\n'
' IO depths : 1=0.3%, 2=6.1%, '
'4=93.6%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1530137,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=854MiB/s (895MB/s), '
'854MiB/s-854MiB/s '
'(895MB/s-895MB/s), io=5977MiB '
'(6267MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_285.output'),
'state': {'begin': 1666727260.9420917,
'cmd': 'cat '
'/tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7870001792907715,
'end': 1666727261.729092,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_285.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_285.state')},
'cmd_286': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_286.output'),
'state': {'begin': 1666727263.7560027,
'cmd': 'rm '
'/tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7789556980133057,
'end': 1666727264.5349584,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_286.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_286.state')},
'cmd_287': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_287.output'),
'state': {'begin': 1666727264.5373862,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.526393175125122,
'end': 1666727276.0637794,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_287.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_287.state')},
'cmd_288': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734365,\n'
' "timestamp_ms" : '
'1666734365443,\n'
' "time" : "Tue Oct 25 23:46:05 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6263152640,\n'
' "io_kbytes" : 6116360,\n'
' "bw_bytes" : 894608290,\n'
' "bw" : 873640,\n'
' "iops" : 218409.798600,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1529087,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1552,\n'
' "max" : 1296060,\n'
' "mean" : 2038.297024,\n'
' "stddev" : '
'1293.969014,\n'
' "N" : 1529087\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 652,\n'
' "max" : 1309395,\n'
' "mean" : '
'15867.342854,\n'
' "stddev" : '
'6045.816309,\n'
' "N" : 1529090,\n'
' "percentile" : {\n'
' "1.000000" : 10304,\n'
' "5.000000" : 10560,\n'
' "10.000000" : '
'12992,\n'
' "20.000000" : '
'13760,\n'
' "30.000000" : '
'14016,\n'
' "40.000000" : '
'14400,\n'
' "50.000000" : '
'14656,\n'
' "60.000000" : '
'14784,\n'
' "70.000000" : '
'15168,\n'
' "80.000000" : '
'16192,\n'
' "90.000000" : '
'18048,\n'
' "95.000000" : '
'27520,\n'
' "99.000000" : '
'38656,\n'
' "99.500000" : '
'42752,\n'
' "99.900000" : '
'78336,\n'
' "99.950000" : '
'90624,\n'
' "99.990000" : '
'103936\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9738,\n'
' "max" : 1311068,\n'
' "mean" : '
'17905.639056,\n'
' "stddev" : '
'6101.138943,\n'
' "N" : 1529090\n'
' },\n'
' "bw_min" : 871208,\n'
' "bw_max" : 876849,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'873847.500000,\n'
' "bw_dev" : 1583.292322,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 217802,\n'
' "iops_max" : 219212,\n'
' "iops_mean" : '
'218461.857143,\n'
' "iops_stddev" : '
'395.786628,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 41.414286,\n'
' "sys_cpu" : 58.557143,\n'
' "ctx" : 45,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.279579,\n'
' "2" : 6.606622,\n'
' "4" : 93.113799,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.020470,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.132170,\n'
' "20" : 92.125039,\n'
' "50" : 7.538289,\n'
' "100" : 0.154994,\n'
' "250" : 0.023413,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=88567: Tue '
'Oct 25 23:46:05 2022\n'
' read: IOPS=218k, BW=853MiB/s '
'(895MB/s)(5973MiB/7001msec)\n'
' slat (nsec): min=1552, '
'max=1296.1k, avg=2038.30, '
'stdev=1293.97\n'
' clat (nsec): min=652, '
'max=1309.4k, avg=15867.34, '
'stdev=6045.82\n'
' lat (usec): min=9, '
'max=1311, avg=17.91, stdev= '
'6.10\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 11], '
'5.00th=[ 11], 10.00th=[ 13], '
'20.00th=[ 14],\n'
' | 30.00th=[ 15], '
'40.00th=[ 15], 50.00th=[ '
'15], 60.00th=[ 15],\n'
' | 70.00th=[ 16], '
'80.00th=[ 17], 90.00th=[ '
'19], 95.00th=[ 28],\n'
' | 99.00th=[ 39], '
'99.50th=[ 43], 99.90th=[ '
'79], 99.95th=[ 91],\n'
' | 99.99th=[ 104]\n'
' bw ( KiB/s): min=871208, '
'max=876849, per=100.00%, '
'avg=873847.50, stdev=1583.29, '
'samples=14\n'
' iops : min=217802, '
'max=219212, avg=218461.86, '
'stdev=395.79, samples=14\n'
' lat (nsec) : 750=0.02%, '
'1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.13%, 20=92.13%, 50=7.54%, '
'100=0.15%\n'
' lat (usec) : 250=0.02%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=41.41%, '
'sys=58.56%, ctx=45, majf=0, '
'minf=0\n'
' IO depths : 1=0.3%, 2=6.6%, '
'4=93.1%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1529087,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=853MiB/s (895MB/s), '
'853MiB/s-853MiB/s '
'(895MB/s-895MB/s), io=5973MiB '
'(6263MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_288.output'),
'state': {'begin': 1666727276.0664513,
'cmd': 'cat '
'/tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7896168231964111,
'end': 1666727276.8560681,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_288.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_288.state')},
'cmd_289': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_libaio-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_289.output'),
'state': {'begin': 1666727277.8519075,
'cmd': 'rm '
'/tmp/fio-output_4K_4_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7837321758270264,
'end': 1666727278.6356397,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_289.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_289.state')},
'cmd_29': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_29.output'),
'state': {'begin': 1666726029.2404494,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring-reference_0.txt"',
'cwd': 'None',
'elapsed': 11.047251462936401,
'end': 1666726040.287701,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_29.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_29.state')},
'cmd_290': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_290.output'),
'state': {'begin': 1666727278.637711,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_libaio-xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.082463502883911,
'end': 1666727289.7201746,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_290.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_290.state')},
'cmd_291': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734379,\n'
' "timestamp_ms" : '
'1666734379201,\n'
' "time" : "Tue Oct 25 23:46:19 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'5072072704,\n'
' "io_kbytes" : 4953196,\n'
' "bw_bytes" : 724478317,\n'
' "bw" : 707498,\n'
' "iops" : 176874.160834,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1238296,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1392,\n'
' "max" : 24916,\n'
' "mean" : 2289.736705,\n'
' "stddev" : '
'1489.650647,\n'
' "N" : 1238296\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2695,\n'
' "max" : 133280,\n'
' "mean" : '
'20010.147292,\n'
' "stddev" : '
'5825.442065,\n'
' "N" : 1238299,\n'
' "percentile" : {\n'
' "1.000000" : 14912,\n'
' "5.000000" : 15680,\n'
' "10.000000" : '
'15808,\n'
' "20.000000" : '
'16064,\n'
' "30.000000" : '
'17280,\n'
' "40.000000" : '
'17536,\n'
' "50.000000" : '
'19328,\n'
' "60.000000" : '
'19584,\n'
' "70.000000" : '
'21376,\n'
' "80.000000" : '
'21888,\n'
' "90.000000" : '
'22144,\n'
' "95.000000" : '
'31360,\n'
' "99.000000" : '
'43776,\n'
' "99.500000" : '
'46848,\n'
' "99.900000" : '
'81408,\n'
' "99.950000" : '
'93696,\n'
' "99.990000" : '
'105984\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10369,\n'
' "max" : 134742,\n'
' "mean" : '
'22299.882204,\n'
' "stddev" : '
'5670.530977,\n'
' "N" : 1238299\n'
' },\n'
' "bw_min" : 703320,\n'
' "bw_max" : 710557,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'707687.785714,\n'
' "bw_dev" : 1699.738812,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 175830,\n'
' "iops_max" : 177639,\n'
' "iops_mean" : '
'176921.928571,\n'
' "iops_stddev" : '
'424.902245,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.414286,\n'
' "sys_cpu" : 50.042857,\n'
' "ctx" : 319314,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.027215,\n'
' "20" : 65.634549,\n'
' "50" : 34.085227,\n'
' "100" : 0.224744,\n'
' "250" : 0.028345,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=88606: Tue '
'Oct 25 23:46:19 2022\n'
' read: IOPS=177k, BW=691MiB/s '
'(724MB/s)(4837MiB/7001msec)\n'
' slat (nsec): min=1392, '
'max=24916, avg=2289.74, '
'stdev=1489.65\n'
' clat (usec): min=2, max=133, '
'avg=20.01, stdev= 5.83\n'
' lat (usec): min=10, '
'max=134, avg=22.30, stdev= 5.67\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 15], '
'5.00th=[ 16], 10.00th=[ 16], '
'20.00th=[ 17],\n'
' | 30.00th=[ 18], '
'40.00th=[ 18], 50.00th=[ '
'20], 60.00th=[ 20],\n'
' | 70.00th=[ 22], '
'80.00th=[ 22], 90.00th=[ '
'23], 95.00th=[ 32],\n'
' | 99.00th=[ 44], '
'99.50th=[ 47], 99.90th=[ '
'82], 99.95th=[ 94],\n'
' | 99.99th=[ 106]\n'
' bw ( KiB/s): min=703320, '
'max=710557, per=100.00%, '
'avg=707687.79, stdev=1699.74, '
'samples=14\n'
' iops : min=175830, '
'max=177639, avg=176921.93, '
'stdev=424.90, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.03%, 20=65.63%, 50=34.09%, '
'100=0.22%\n'
' lat (usec) : 250=0.03%\n'
' cpu : usr=15.41%, '
'sys=50.04%, ctx=319314, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1238296,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=691MiB/s (724MB/s), '
'691MiB/s-691MiB/s '
'(724MB/s-724MB/s), io=4837MiB '
'(5072MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_291.output'),
'state': {'begin': 1666727289.723327,
'cmd': 'cat '
'/tmp/fio-output_4K_4_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.789381742477417,
'end': 1666727290.5127087,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_291.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_291.state')},
'cmd_292': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_libaio-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_292.output'),
'state': {'begin': 1666727291.4978764,
'cmd': 'rm '
'/tmp/fio-output_4K_4_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7889368534088135,
'end': 1666727292.2868133,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_292.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_292.state')},
'cmd_293': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_293.output'),
'state': {'begin': 1666727292.289012,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_libaio-xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.090299606323242,
'end': 1666727303.3793116,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_293.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_293.state')},
'cmd_294': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734392,\n'
' "timestamp_ms" : '
'1666734392857,\n'
' "time" : "Tue Oct 25 23:46:32 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'5079027712,\n'
' "io_kbytes" : 4959988,\n'
' "bw_bytes" : 725471748,\n'
' "bw" : 708468,\n'
' "iops" : 177116.554778,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1239993,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1432,\n'
' "max" : 25278,\n'
' "mean" : 2296.353298,\n'
' "stddev" : '
'1460.013791,\n'
' "N" : 1239994\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5000,\n'
' "max" : 315411,\n'
' "mean" : '
'19955.888521,\n'
' "stddev" : '
'5758.486187,\n'
' "N" : 1239996,\n'
' "percentile" : {\n'
' "1.000000" : 14656,\n'
' "5.000000" : 15808,\n'
' "10.000000" : '
'15808,\n'
' "20.000000" : '
'16064,\n'
' "30.000000" : '
'17280,\n'
' "40.000000" : '
'17536,\n'
' "50.000000" : '
'19328,\n'
' "60.000000" : '
'19584,\n'
' "70.000000" : '
'21376,\n'
' "80.000000" : '
'21632,\n'
' "90.000000" : '
'21888,\n'
' "95.000000" : '
'31360,\n'
' "99.000000" : '
'43264,\n'
' "99.500000" : '
'46336,\n'
' "99.900000" : '
'76288,\n'
' "99.950000" : '
'92672,\n'
' "99.990000" : '
'105984\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10500,\n'
' "max" : 316904,\n'
' "mean" : '
'22252.243399,\n'
' "stddev" : '
'5586.706939,\n'
' "N" : 1239996\n'
' },\n'
' "bw_min" : 704384,\n'
' "bw_max" : 710608,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'708637.857143,\n'
' "bw_dev" : 1613.810486,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 176096,\n'
' "iops_max" : 177652,\n'
' "iops_mean" : '
'177159.285714,\n'
' "iops_stddev" : '
'403.671311,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 16.328571,\n'
' "sys_cpu" : 49.528571,\n'
' "ctx" : 320288,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.015161,\n'
' "20" : 66.261019,\n'
' "50" : 33.495592,\n'
' "100" : 0.202824,\n'
' "250" : 0.025323,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=88647: Tue '
'Oct 25 23:46:32 2022\n'
' read: IOPS=177k, BW=692MiB/s '
'(725MB/s)(4844MiB/7001msec)\n'
' slat (nsec): min=1432, '
'max=25278, avg=2296.35, '
'stdev=1460.01\n'
' clat (usec): min=5, max=315, '
'avg=19.96, stdev= 5.76\n'
' lat (usec): min=10, '
'max=316, avg=22.25, stdev= 5.59\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 15], '
'5.00th=[ 16], 10.00th=[ 16], '
'20.00th=[ 17],\n'
' | 30.00th=[ 18], '
'40.00th=[ 18], 50.00th=[ '
'20], 60.00th=[ 20],\n'
' | 70.00th=[ 22], '
'80.00th=[ 22], 90.00th=[ '
'22], 95.00th=[ 32],\n'
' | 99.00th=[ 44], '
'99.50th=[ 47], 99.90th=[ '
'77], 99.95th=[ 93],\n'
' | 99.99th=[ 106]\n'
' bw ( KiB/s): min=704384, '
'max=710608, per=100.00%, '
'avg=708637.86, stdev=1613.81, '
'samples=14\n'
' iops : min=176096, '
'max=177652, avg=177159.29, '
'stdev=403.67, samples=14\n'
' lat (usec) : 10=0.02%, '
'20=66.26%, 50=33.50%, 100=0.20%, '
'250=0.03%\n'
' lat (usec) : 500=0.01%\n'
' cpu : usr=16.33%, '
'sys=49.53%, ctx=320288, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1239993,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=692MiB/s (725MB/s), '
'692MiB/s-692MiB/s '
'(725MB/s-725MB/s), io=4844MiB '
'(5079MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_294.output'),
'state': {'begin': 1666727303.386083,
'cmd': 'cat '
'/tmp/fio-output_4K_4_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8056910037994385,
'end': 1666727304.191774,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_294.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_294.state')},
'cmd_295': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_4_libaio-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_295.output'),
'state': {'begin': 1666727305.191382,
'cmd': 'rm '
'/tmp/fio-output_4K_4_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7836463451385498,
'end': 1666727305.9750283,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_295.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_295.state')},
'cmd_296': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_296.output'),
'state': {'begin': 1666727305.977193,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_4_libaio-xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.102250337600708,
'end': 1666727317.0794435,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_296.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_296.state')},
'cmd_297': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734406,\n'
' "timestamp_ms" : '
'1666734406559,\n'
' "time" : "Tue Oct 25 23:46:46 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'5081395200,\n'
' "io_kbytes" : 4962300,\n'
' "bw_bytes" : 725809912,\n'
' "bw" : 708798,\n'
' "iops" : 177199.114412,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1240571,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1402,\n'
' "max" : 30307,\n'
' "mean" : 2212.269113,\n'
' "stddev" : '
'1350.794632,\n'
' "N" : 1240572\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2756,\n'
' "max" : 142718,\n'
' "mean" : '
'20076.305615,\n'
' "stddev" : '
'5924.159167,\n'
' "N" : 1240574,\n'
' "percentile" : {\n'
' "1.000000" : 14144,\n'
' "5.000000" : 14784,\n'
' "10.000000" : '
'15040,\n'
' "20.000000" : '
'15936,\n'
' "30.000000" : '
'17280,\n'
' "40.000000" : '
'17536,\n'
' "50.000000" : '
'19328,\n'
' "60.000000" : '
'21376,\n'
' "70.000000" : '
'21632,\n'
' "80.000000" : '
'21888,\n'
' "90.000000" : '
'22144,\n'
' "95.000000" : '
'31360,\n'
' "99.000000" : '
'43776,\n'
' "99.500000" : '
'46336,\n'
' "99.900000" : '
'80384,\n'
' "99.950000" : '
'94720,\n'
' "99.990000" : '
'105984\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10880,\n'
' "max" : 144201,\n'
' "mean" : '
'22288.576315,\n'
' "stddev" : '
'6010.338472,\n'
' "N" : 1240574\n'
' },\n'
' "bw_min" : 700656,\n'
' "bw_max" : 710264,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'708977.357143,\n'
' "bw_dev" : 2422.535722,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 175164,\n'
' "iops_max" : 177566,\n'
' "iops_mean" : '
'177244.428571,\n'
' "iops_stddev" : '
'605.618910,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.228571,\n'
' "sys_cpu" : 50.371429,\n'
' "ctx" : 318778,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.026842,\n'
' "20" : 57.241706,\n'
' "50" : 42.487371,\n'
' "100" : 0.212402,\n'
' "250" : 0.031760,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=88686: Tue '
'Oct 25 23:46:46 2022\n'
' read: IOPS=177k, BW=692MiB/s '
'(726MB/s)(4846MiB/7001msec)\n'
' slat (nsec): min=1402, '
'max=30307, avg=2212.27, '
'stdev=1350.79\n'
' clat (usec): min=2, max=142, '
'avg=20.08, stdev= 5.92\n'
' lat (usec): min=10, '
'max=144, avg=22.29, stdev= 6.01\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 15], '
'5.00th=[ 15], 10.00th=[ 16], '
'20.00th=[ 16],\n'
' | 30.00th=[ 18], '
'40.00th=[ 18], 50.00th=[ '
'20], 60.00th=[ 22],\n'
' | 70.00th=[ 22], '
'80.00th=[ 22], 90.00th=[ '
'23], 95.00th=[ 32],\n'
' | 99.00th=[ 44], '
'99.50th=[ 47], 99.90th=[ '
'81], 99.95th=[ 95],\n'
' | 99.99th=[ 106]\n'
' bw ( KiB/s): min=700656, '
'max=710264, per=100.00%, '
'avg=708977.36, stdev=2422.54, '
'samples=14\n'
' iops : min=175164, '
'max=177566, avg=177244.43, '
'stdev=605.62, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.03%, 20=57.24%, 50=42.49%, '
'100=0.21%\n'
' lat (usec) : 250=0.03%\n'
' cpu : usr=14.23%, '
'sys=50.37%, ctx=318778, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1240571,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=692MiB/s (726MB/s), '
'692MiB/s-692MiB/s '
'(726MB/s-726MB/s), io=4846MiB '
'(5081MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_297.output'),
'state': {'begin': 1666727317.082374,
'cmd': 'cat '
'/tmp/fio-output_4K_4_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7876636981964111,
'end': 1666727317.8700378,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_297.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_297.state')},
'cmd_298': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_298.output'),
'state': {'begin': 1666727318.8601072,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.793532133102417,
'end': 1666727319.6536393,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_298.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_298.state')},
'cmd_299': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_299.output'),
'state': {'begin': 1666727319.6556458,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt"',
'cwd': 'None',
'elapsed': 11.054964542388916,
'end': 1666727330.7106104,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_299.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_299.state')},
'cmd_30': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=io_uring, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733129,\n'
' "timestamp_ms" : '
'1666733129789,\n'
' "time" : "Tue Oct 25 23:25:29 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "io_uring",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1804812288,\n'
' "io_kbytes" : 1762512,\n'
' "bw_bytes" : 257793499,\n'
' "bw" : 251751,\n'
' "iops" : 62937.723182,\n'
' "runtime" : 7001,\n'
' "total_ios" : 440627,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2425,\n'
' "max" : 42410,\n'
' "mean" : 2632.123117,\n'
' "stddev" : 277.348007,\n'
' "N" : 440628\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 210,\n'
' "max" : 185869,\n'
' "mean" : 12588.789237,\n'
' "stddev" : '
'2063.603963,\n'
' "N" : 440627,\n'
' "percentile" : {\n'
' "1.000000" : 12096,\n'
' "5.000000" : 12224,\n'
' "10.000000" : 12224,\n'
' "20.000000" : 12224,\n'
' "30.000000" : 12224,\n'
' "40.000000" : 12352,\n'
' "50.000000" : 12352,\n'
' "60.000000" : 12352,\n'
' "70.000000" : 12352,\n'
' "80.000000" : 12352,\n'
' "90.000000" : 12480,\n'
' "95.000000" : 12608,\n'
' "99.000000" : 22400,\n'
' "99.500000" : 28032,\n'
' "99.900000" : 39168,\n'
' "99.950000" : 50432,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12183,\n'
' "max" : 188574,\n'
' "mean" : 15220.912393,\n'
' "stddev" : '
'2084.834946,\n'
' "N" : 440627\n'
' },\n'
' "bw_min" : 250536,\n'
' "bw_max" : 252744,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'251847.142857,\n'
' "bw_dev" : 664.546445,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 62634,\n'
' "iops_max" : 63186,\n'
' "iops_mean" : '
'62961.714286,\n'
' "iops_stddev" : '
'166.188229,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 9.728571,\n'
' "sys_cpu" : 38.085714,\n'
' "ctx" : 440609,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 98.589737,\n'
' "50" : 1.345809,\n'
' "100" : 0.053787,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 630140,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 6063,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 6063,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=84772: Tue '
'Oct 25 23:25:29 2022\n'
' read: IOPS=62.9k, BW=246MiB/s '
'(258MB/s)(1721MiB/7001msec)\n'
' slat (nsec): min=2425, '
'max=42410, avg=2632.12, '
'stdev=277.35\n'
' clat (nsec): min=210, '
'max=185869, avg=12588.79, '
'stdev=2063.60\n'
' lat (usec): min=12, max=188, '
'avg=15.22, stdev= 2.08\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12096], '
'5.00th=[12224], 10.00th=[12224], '
'20.00th=[12224],\n'
' | 30.00th=[12224], '
'40.00th=[12352], 50.00th=[12352], '
'60.00th=[12352],\n'
' | 70.00th=[12352], '
'80.00th=[12352], 90.00th=[12480], '
'95.00th=[12608],\n'
' | 99.00th=[22400], '
'99.50th=[28032], 99.90th=[39168], '
'99.95th=[50432],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=250536, '
'max=252744, per=100.00%, '
'avg=251847.14, stdev=664.55, '
'samples=14\n'
' iops : min=62634, '
'max=63186, avg=62961.71, '
'stdev=166.19, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=98.59%, 50=1.35%, '
'100=0.05%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=9.73%, '
'sys=38.09%, ctx=440609, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=440627,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=246MiB/s (258MB/s), '
'246MiB/s-246MiB/s '
'(258MB/s-258MB/s), io=1721MiB '
'(1805MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=630140/0, '
'merge=0/0, ticks=6063/0, '
'in_queue=6063, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_30.output'),
'state': {'begin': 1666726040.2903283,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7855043411254883,
'end': 1666726041.0758326,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_30.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_30.state')},
'cmd_300': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, '
'ioengine=io_uring_cmd, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734420,\n'
' "timestamp_ms" : '
'1666734420189,\n'
' "time" : "Tue Oct 25 23:47:00 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6420176896,\n'
' "io_kbytes" : 6269704,\n'
' "bw_bytes" : 917037122,\n'
' "bw" : 895544,\n'
' "iops" : 223884.873589,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567418,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1082,\n'
' "max" : 24687,\n'
' "mean" : 1805.728822,\n'
' "stddev" : '
'832.327889,\n'
' "N" : 1567419\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 11222,\n'
' "max" : 215775,\n'
' "mean" : '
'33639.284363,\n'
' "stddev" : '
'8658.611942,\n'
' "N" : 1567425,\n'
' "percentile" : {\n'
' "1.000000" : 24960,\n'
' "5.000000" : 28800,\n'
' "10.000000" : '
'29568,\n'
' "20.000000" : '
'30080,\n'
' "30.000000" : '
'30080,\n'
' "40.000000" : '
'30848,\n'
' "50.000000" : '
'31360,\n'
' "60.000000" : '
'31616,\n'
' "70.000000" : '
'31872,\n'
' "80.000000" : '
'32640,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'52480,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'127488,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 15389,\n'
' "max" : 218299,\n'
' "mean" : '
'35445.013616,\n'
' "stddev" : '
'8573.177387,\n'
' "N" : 1567425\n'
' },\n'
' "bw_min" : 893760,\n'
' "bw_max" : 898008,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895765.357143,\n'
' "bw_dev" : 1348.294627,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223440,\n'
' "iops_max" : 224500,\n'
' "iops_mean" : '
'223941.285714,\n'
' "iops_stddev" : '
'336.789920,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.514286,\n'
' "sys_cpu" : 58.171429,\n'
' "ctx" : 756092,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999936,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.102462,\n'
' "50" : 91.998178,\n'
' "100" : 7.644419,\n'
' "250" : 0.255388,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=88726: Tue Oct 25 23:47:00 '
'2022\n'
' read: IOPS=224k, BW=875MiB/s '
'(917MB/s)(6123MiB/7001msec)\n'
' slat (nsec): min=1082, '
'max=24687, avg=1805.73, '
'stdev=832.33\n'
' clat (usec): min=11, '
'max=215, avg=33.64, stdev= 8.66\n'
' lat (usec): min=15, '
'max=218, avg=35.45, stdev= 8.57\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 29], 10.00th=[ 30], '
'20.00th=[ 31],\n'
' | 30.00th=[ 31], '
'40.00th=[ 31], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 32], '
'80.00th=[ 33], 90.00th=[ '
'46], 95.00th=[ 53],\n'
' | 99.00th=[ 63], '
'99.50th=[ 68], 99.90th=[ '
'128], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=893760, '
'max=898008, per=100.00%, '
'avg=895765.36, stdev=1348.29, '
'samples=14\n'
' iops : min=223440, '
'max=224500, avg=223941.29, '
'stdev=336.79, samples=14\n'
' lat (usec) : 20=0.10%, '
'50=92.00%, 100=7.64%, 250=0.26%\n'
' cpu : usr=14.51%, '
'sys=58.17%, ctx=756092, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567418,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=875MiB/s (917MB/s), '
'875MiB/s-875MiB/s '
'(917MB/s-917MB/s), io=6123MiB '
'(6420MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_300.output'),
'state': {'begin': 1666727330.713518,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.781747579574585,
'end': 1666727331.4952655,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_300.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_300.state')},
'cmd_301': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_301.output'),
'state': {'begin': 1666727332.520811,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.785956621170044,
'end': 1666727333.3067677,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_301.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_301.state')},
'cmd_302': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_302.output'),
'state': {'begin': 1666727333.309005,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt"',
'cwd': 'None',
'elapsed': 11.057814359664917,
'end': 1666727344.3668194,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_302.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_302.state')},
'cmd_303': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, '
'ioengine=io_uring_cmd, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734433,\n'
' "timestamp_ms" : '
'1666734433845,\n'
' "time" : "Tue Oct 25 23:47:13 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6416601088,\n'
' "io_kbytes" : 6266212,\n'
' "bw_bytes" : 916526365,\n'
' "bw" : 895045,\n'
' "iops" : 223760.319954,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1566546,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1102,\n'
' "max" : 42740,\n'
' "mean" : 1833.665654,\n'
' "stddev" : '
'837.174942,\n'
' "N" : 1566546\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4929,\n'
' "max" : 212439,\n'
' "mean" : '
'33633.179418,\n'
' "stddev" : '
'8764.799429,\n'
' "N" : 1566553,\n'
' "percentile" : {\n'
' "1.000000" : 24704,\n'
' "5.000000" : 28800,\n'
' "10.000000" : '
'29568,\n'
' "20.000000" : '
'30080,\n'
' "30.000000" : '
'30080,\n'
' "40.000000" : '
'30592,\n'
' "50.000000" : '
'31360,\n'
' "60.000000" : '
'31616,\n'
' "70.000000" : '
'31872,\n'
' "80.000000" : '
'32640,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'52480,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'68096,\n'
' "99.900000" : '
'130560,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 15479,\n'
' "max" : 215695,\n'
' "mean" : '
'35466.844208,\n'
' "stddev" : '
'8680.017687,\n'
' "N" : 1566553\n'
' },\n'
' "bw_min" : 891968,\n'
' "bw_max" : 897104,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895253.928571,\n'
' "bw_dev" : 1517.357950,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 222992,\n'
' "iops_max" : 224276,\n'
' "iops_mean" : '
'223813.428571,\n'
' "iops_stddev" : '
'379.390141,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.785714,\n'
' "sys_cpu" : 58.928571,\n'
' "ctx" : 753694,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999936,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 0.106285,\n'
' "50" : 92.040195,\n'
' "100" : 7.578456,\n'
' "250" : 0.275447,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=88764: Tue Oct 25 23:47:13 '
'2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6119MiB/7001msec)\n'
' slat (nsec): min=1102, '
'max=42740, avg=1833.67, '
'stdev=837.17\n'
' clat (usec): min=4, max=212, '
'avg=33.63, stdev= 8.76\n'
' lat (usec): min=15, '
'max=215, avg=35.47, stdev= 8.68\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 29], 10.00th=[ 30], '
'20.00th=[ 31],\n'
' | 30.00th=[ 31], '
'40.00th=[ 31], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 32], '
'80.00th=[ 33], 90.00th=[ '
'46], 95.00th=[ 53],\n'
' | 99.00th=[ 63], '
'99.50th=[ 69], 99.90th=[ '
'131], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=891968, '
'max=897104, per=100.00%, '
'avg=895253.93, stdev=1517.36, '
'samples=14\n'
' iops : min=222992, '
'max=224276, avg=223813.43, '
'stdev=379.39, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=0.11%, 50=92.04%, 100=7.58%, '
'250=0.28%\n'
' cpu : usr=13.79%, '
'sys=58.93%, ctx=753694, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1566546,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6119MiB '
'(6417MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_303.output'),
'state': {'begin': 1666727344.3698432,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.785027027130127,
'end': 1666727345.1548703,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_303.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_303.state')},
'cmd_304': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_304.output'),
'state': {'begin': 1666727346.194987,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7897701263427734,
'end': 1666727346.9847572,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_304.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_304.state')},
'cmd_305': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_305.output'),
'state': {'begin': 1666727346.9870663,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt"',
'cwd': 'None',
'elapsed': 11.042422533035278,
'end': 1666727358.0294888,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_305.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_305.state')},
'cmd_306': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, '
'ioengine=io_uring_cmd, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734447,\n'
' "timestamp_ms" : '
'1666734447509,\n'
' "time" : "Tue Oct 25 23:47:27 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6417027072,\n'
' "io_kbytes" : 6266628,\n'
' "bw_bytes" : 916587212,\n'
' "bw" : 895104,\n'
' "iops" : 223775.032138,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1566649,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1082,\n'
' "max" : 28684,\n'
' "mean" : 1801.328699,\n'
' "stddev" : '
'837.999667,\n'
' "N" : 1566650\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 11131,\n'
' "max" : 206056,\n'
' "mean" : '
'33649.558555,\n'
' "stddev" : '
'8666.871644,\n'
' "N" : 1566656,\n'
' "percentile" : {\n'
' "1.000000" : 24704,\n'
' "5.000000" : 28800,\n'
' "10.000000" : '
'29824,\n'
' "20.000000" : '
'30080,\n'
' "30.000000" : '
'30336,\n'
' "40.000000" : '
'30848,\n'
' "50.000000" : '
'31616,\n'
' "60.000000" : '
'31616,\n'
' "70.000000" : '
'31872,\n'
' "80.000000" : '
'32640,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'128512,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 15359,\n'
' "max" : 208260,\n'
' "mean" : '
'35450.887665,\n'
' "stddev" : '
'8587.323029,\n'
' "N" : 1566656\n'
' },\n'
' "bw_min" : 885416,\n'
' "bw_max" : 897224,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895327.500000,\n'
' "bw_dev" : 3021.324589,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 221354,\n'
' "iops_max" : 224306,\n'
' "iops_mean" : '
'223831.857143,\n'
' "iops_stddev" : '
'755.321415,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.185714,\n'
' "sys_cpu" : 58.757143,\n'
' "ctx" : 755492,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999936,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.092554,\n'
' "50" : 92.064145,\n'
' "100" : 7.584341,\n'
' "250" : 0.259407,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=88804: Tue Oct 25 23:47:27 '
'2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6120MiB/7001msec)\n'
' slat (nsec): min=1082, '
'max=28684, avg=1801.33, '
'stdev=838.00\n'
' clat (usec): min=11, '
'max=206, avg=33.65, stdev= 8.67\n'
' lat (usec): min=15, '
'max=208, avg=35.45, stdev= 8.59\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 29], 10.00th=[ 30], '
'20.00th=[ 31],\n'
' | 30.00th=[ 31], '
'40.00th=[ 31], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 32], '
'80.00th=[ 33], 90.00th=[ '
'46], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 68], 99.90th=[ '
'129], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=885416, '
'max=897224, per=100.00%, '
'avg=895327.50, stdev=3021.32, '
'samples=14\n'
' iops : min=221354, '
'max=224306, avg=223831.86, '
'stdev=755.32, samples=14\n'
' lat (usec) : 20=0.09%, '
'50=92.06%, 100=7.58%, 250=0.26%\n'
' cpu : usr=14.19%, '
'sys=58.76%, ctx=755492, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1566649,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6120MiB '
'(6417MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_306.output'),
'state': {'begin': 1666727358.0324516,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7928454875946045,
'end': 1666727358.825297,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_306.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_306.state')},
'cmd_307': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_307.output'),
'state': {'begin': 1666727359.870337,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7896568775177002,
'end': 1666727360.659994,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_307.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_307.state')},
'cmd_308': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_308.output'),
'state': {'begin': 1666727360.6621885,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.059838771820068,
'end': 1666727371.7220273,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_308.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_308.state')},
'cmd_309': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734461,\n'
' "timestamp_ms" : '
'1666734461202,\n'
' "time" : "Tue Oct 25 23:47:41 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6421319680,\n'
' "io_kbytes" : 6270820,\n'
' "bw_bytes" : 917200354,\n'
' "bw" : 895703,\n'
' "iops" : 223924.725039,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567697,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1152,\n'
' "max" : 175619,\n'
' "mean" : 1747.089774,\n'
' "stddev" : '
'791.058747,\n'
' "N" : 1567698\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2465,\n'
' "max" : 215894,\n'
' "mean" : '
'33673.439684,\n'
' "stddev" : '
'8679.324358,\n'
' "N" : 1567704,\n'
' "percentile" : {\n'
' "1.000000" : 24960,\n'
' "5.000000" : 28800,\n'
' "10.000000" : '
'29056,\n'
' "20.000000" : '
'29824,\n'
' "30.000000" : '
'30336,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31616,\n'
' "60.000000" : '
'31872,\n'
' "70.000000" : '
'32384,\n'
' "80.000000" : '
'33024,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'66048,\n'
' "99.900000" : '
'129536,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'158720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 15198,\n'
' "max" : 217137,\n'
' "mean" : '
'35420.530389,\n'
' "stddev" : '
'8614.100339,\n'
' "N" : 1567704\n'
' },\n'
' "bw_min" : 894104,\n'
' "bw_max" : 897792,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895917.571429,\n'
' "bw_dev" : 1033.548683,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223526,\n'
' "iops_max" : 224448,\n'
' "iops_mean" : '
'223979.500000,\n'
' "iops_stddev" : '
'258.444087,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.142857,\n'
' "sys_cpu" : 56.985714,\n'
' "ctx" : 712649,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 0.070932,\n'
' "50" : 92.202383,\n'
' "100" : 7.466940,\n'
' "250" : 0.259999,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=88843: Tue '
'Oct 25 23:47:41 2022\n'
' read: IOPS=224k, BW=875MiB/s '
'(917MB/s)(6124MiB/7001msec)\n'
' slat (nsec): min=1152, '
'max=175619, avg=1747.09, '
'stdev=791.06\n'
' clat (usec): min=2, max=215, '
'avg=33.67, stdev= 8.68\n'
' lat (usec): min=15, '
'max=217, avg=35.42, stdev= 8.61\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 29], 10.00th=[ 30], '
'20.00th=[ 30],\n'
' | 30.00th=[ 31], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 33], '
'80.00th=[ 34], 90.00th=[ '
'46], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 67], 99.90th=[ '
'130], 99.95th=[ 141],\n'
' | 99.99th=[ 159]\n'
' bw ( KiB/s): min=894104, '
'max=897792, per=100.00%, '
'avg=895917.57, stdev=1033.55, '
'samples=14\n'
' iops : min=223526, '
'max=224448, avg=223979.50, '
'stdev=258.44, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=0.07%, 50=92.20%, '
'100=7.47%\n'
' lat (usec) : 250=0.26%\n'
' cpu : usr=17.14%, '
'sys=56.99%, ctx=712649, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567697,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=875MiB/s (917MB/s), '
'875MiB/s-875MiB/s '
'(917MB/s-917MB/s), io=6124MiB '
'(6421MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_309.output'),
'state': {'begin': 1666727371.7245438,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7898790836334229,
'end': 1666727372.514423,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_309.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_309.state')},
'cmd_31': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_31.output'),
'state': {'begin': 1666726042.0663688,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7834937572479248,
'end': 1666726042.8498626,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_31.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_31.state')},
'cmd_310': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_310.output'),
'state': {'begin': 1666727373.5050867,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7769813537597656,
'end': 1666727374.282068,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_310.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_310.state')},
'cmd_311': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_311.output'),
'state': {'begin': 1666727374.2836246,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.051728963851929,
'end': 1666727385.3353536,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_311.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_311.state')},
'cmd_312': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734474,\n'
' "timestamp_ms" : '
'1666734474814,\n'
' "time" : "Tue Oct 25 23:47:54 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6416293888,\n'
' "io_kbytes" : 6265912,\n'
' "bw_bytes" : 916482486,\n'
' "bw" : 895002,\n'
' "iops" : 223749.607199,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1566471,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1192,\n'
' "max" : 40917,\n'
' "mean" : 1637.510650,\n'
' "stddev" : '
'539.612945,\n'
' "N" : 1566471\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 221,\n'
' "max" : 212459,\n'
' "mean" : '
'33807.519823,\n'
' "stddev" : '
'8996.658594,\n'
' "N" : 1566478,\n'
' "percentile" : {\n'
' "1.000000" : 23936,\n'
' "5.000000" : 27520,\n'
' "10.000000" : '
'28800,\n'
' "20.000000" : '
'29568,\n'
' "30.000000" : '
'30592,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31616,\n'
' "60.000000" : '
'32128,\n'
' "70.000000" : '
'32640,\n'
' "80.000000" : '
'33536,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'52480,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'68096,\n'
' "99.900000" : '
'134144,\n'
' "99.950000" : '
'144384,\n'
' "99.990000" : '
'160768\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 14056,\n'
' "max" : 213731,\n'
' "mean" : '
'35445.030097,\n'
' "stddev" : '
'8951.929801,\n'
' "N" : 1566478\n'
' },\n'
' "bw_min" : 888056,\n'
' "bw_max" : 897410,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895350.500000,\n'
' "bw_dev" : 2293.493498,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 222014,\n'
' "iops_max" : 224352,\n'
' "iops_mean" : '
'223837.571429,\n'
' "iops_stddev" : '
'573.327228,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.571429,\n'
' "sys_cpu" : 55.157143,\n'
' "ctx" : 696395,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 0.234029,\n'
' "50" : 91.735755,\n'
' "100" : 7.758267,\n'
' "250" : 0.272204,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=88881: Tue '
'Oct 25 23:47:54 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(916MB/s)(6119MiB/7001msec)\n'
' slat (nsec): min=1192, '
'max=40917, avg=1637.51, '
'stdev=539.61\n'
' clat (nsec): min=221, '
'max=212459, avg=33807.52, '
'stdev=8996.66\n'
' lat (usec): min=14, '
'max=213, avg=35.45, stdev= 8.95\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 24], '
'5.00th=[ 28], 10.00th=[ 29], '
'20.00th=[ 30],\n'
' | 30.00th=[ 31], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 33],\n'
' | 70.00th=[ 33], '
'80.00th=[ 34], 90.00th=[ '
'46], 95.00th=[ 53],\n'
' | 99.00th=[ 63], '
'99.50th=[ 69], 99.90th=[ '
'135], 99.95th=[ 145],\n'
' | 99.99th=[ 161]\n'
' bw ( KiB/s): min=888056, '
'max=897410, per=100.00%, '
'avg=895350.50, stdev=2293.49, '
'samples=14\n'
' iops : min=222014, '
'max=224352, avg=223837.57, '
'stdev=573.33, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=0.23%, 50=91.74%, '
'100=7.76%\n'
' lat (usec) : 250=0.27%\n'
' cpu : usr=17.57%, '
'sys=55.16%, ctx=696395, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1566471,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (916MB/s), '
'874MiB/s-874MiB/s '
'(916MB/s-916MB/s), io=6119MiB '
'(6416MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_312.output'),
'state': {'begin': 1666727385.3380225,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7821457386016846,
'end': 1666727386.1201682,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_312.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_312.state')},
'cmd_313': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_313.output'),
'state': {'begin': 1666727387.1176753,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7792088985443115,
'end': 1666727387.8968842,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_313.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_313.state')},
'cmd_314': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_314.output'),
'state': {'begin': 1666727387.898326,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.055521011352539,
'end': 1666727398.953847,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_314.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_314.state')},
'cmd_315': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734488,\n'
' "timestamp_ms" : '
'1666734488432,\n'
' "time" : "Tue Oct 25 23:48:08 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6412873728,\n'
' "io_kbytes" : 6262572,\n'
' "bw_bytes" : 915993962,\n'
' "bw" : 894525,\n'
' "iops" : 223630.195686,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1565635,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1132,\n'
' "max" : 29926,\n'
' "mean" : 1678.465054,\n'
' "stddev" : '
'744.939201,\n'
' "N" : 1565636\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 7093,\n'
' "max" : 395762,\n'
' "mean" : '
'33756.134564,\n'
' "stddev" : '
'8803.715628,\n'
' "N" : 1565642,\n'
' "percentile" : {\n'
' "1.000000" : 24960,\n'
' "5.000000" : 28800,\n'
' "10.000000" : '
'29056,\n'
' "20.000000" : '
'30080,\n'
' "30.000000" : '
'30592,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31616,\n'
' "60.000000" : '
'31616,\n'
' "70.000000" : '
'32128,\n'
' "80.000000" : '
'33024,\n'
' "90.000000" : '
'45824,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'68096,\n'
' "99.900000" : '
'132096,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 15138,\n'
' "max" : 398527,\n'
' "mean" : '
'35434.600653,\n'
' "stddev" : '
'8758.561285,\n'
' "N" : 1565642\n'
' },\n'
' "bw_min" : 888520,\n'
' "bw_max" : 898120,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'894740.571429,\n'
' "bw_dev" : 2660.843987,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 222130,\n'
' "iops_max" : 224530,\n'
' "iops_mean" : '
'223685.142857,\n'
' "iops_stddev" : '
'665.288700,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.742857,\n'
' "sys_cpu" : 55.314286,\n'
' "ctx" : 725537,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 0.082842,\n'
' "50" : 92.137439,\n'
' "100" : 7.499130,\n'
' "250" : 0.280461,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=88919: Tue '
'Oct 25 23:48:08 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(916MB/s)(6116MiB/7001msec)\n'
' slat (nsec): min=1132, '
'max=29926, avg=1678.47, '
'stdev=744.94\n'
' clat (usec): min=7, max=395, '
'avg=33.76, stdev= 8.80\n'
' lat (usec): min=15, '
'max=398, avg=35.43, stdev= 8.76\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 29], 10.00th=[ 30], '
'20.00th=[ 31],\n'
' | 30.00th=[ 31], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 33], '
'80.00th=[ 34], 90.00th=[ '
'46], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 69], 99.90th=[ '
'133], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=888520, '
'max=898120, per=100.00%, '
'avg=894740.57, stdev=2660.84, '
'samples=14\n'
' iops : min=222130, '
'max=224530, avg=223685.14, '
'stdev=665.29, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=0.08%, 50=92.14%, 100=7.50%, '
'250=0.28%\n'
' lat (usec) : 500=0.01%\n'
' cpu : usr=17.74%, '
'sys=55.31%, ctx=725537, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1565635,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (916MB/s), '
'874MiB/s-874MiB/s '
'(916MB/s-916MB/s), io=6116MiB '
'(6413MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_315.output'),
'state': {'begin': 1666727398.9567535,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7833278179168701,
'end': 1666727399.7400813,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_315.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_315.state')},
'cmd_316': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_316.output'),
'state': {'begin': 1666727401.7916675,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.787926435470581,
'end': 1666727402.579594,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_316.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_316.state')},
'cmd_317': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_317.output'),
'state': {'begin': 1666727402.58144,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.487804651260376,
'end': 1666727414.0692446,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_317.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_317.state')},
'cmd_318': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734503,\n'
' "timestamp_ms" : '
'1666734503475,\n'
' "time" : "Tue Oct 25 23:48:23 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6421094400,\n'
' "io_kbytes" : 6270600,\n'
' "bw_bytes" : 917168175,\n'
' "bw" : 895672,\n'
' "iops" : 223917.011855,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567643,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 661,\n'
' "max" : 38231,\n'
' "mean" : 991.606545,\n'
' "stddev" : '
'179.577583,\n'
' "N" : 1567643\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1854,\n'
' "max" : 230002,\n'
' "mean" : '
'34445.715295,\n'
' "stddev" : '
'8475.141091,\n'
' "N" : 1567649,\n'
' "percentile" : {\n'
' "1.000000" : 28032,\n'
' "5.000000" : 30080,\n'
' "10.000000" : '
'30592,\n'
' "20.000000" : '
'31104,\n'
' "30.000000" : '
'31360,\n'
' "40.000000" : '
'31616,\n'
' "50.000000" : '
'31872,\n'
' "60.000000" : '
'32384,\n'
' "70.000000" : '
'32640,\n'
' "80.000000" : '
'33536,\n'
' "90.000000" : '
'44288,\n'
' "95.000000" : '
'52480,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'66048,\n'
' "99.900000" : '
'129536,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 13204,\n'
' "max" : 230923,\n'
' "mean" : '
'35437.321713,\n'
' "stddev" : '
'8475.876232,\n'
' "N" : 1567649\n'
' },\n'
' "bw_min" : 893891,\n'
' "bw_max" : 897088,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895900.214286,\n'
' "bw_dev" : 990.139709,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223472,\n'
' "iops_max" : 224272,\n'
' "iops_mean" : '
'223975.000000,\n'
' "iops_stddev" : '
'247.652051,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 18,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.100000,\n'
' "4" : 0.720955,\n'
' "8" : 99.261694,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.995006,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 0.022008,\n'
' "50" : 91.171332,\n'
' "100" : 8.551245,\n'
' "250" : 0.255160,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=88970: Tue Oct 25 23:48:23 '
'2022\n'
' read: IOPS=224k, BW=875MiB/s '
'(917MB/s)(6124MiB/7001msec)\n'
' slat (nsec): min=661, '
'max=38231, avg=991.61, '
'stdev=179.58\n'
' clat (nsec): min=1854, '
'max=230002, avg=34445.72, '
'stdev=8475.14\n'
' lat (usec): min=13, '
'max=230, avg=35.44, stdev= 8.48\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 29], '
'5.00th=[ 31], 10.00th=[ 31], '
'20.00th=[ 32],\n'
' | 30.00th=[ 32], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 33],\n'
' | 70.00th=[ 33], '
'80.00th=[ 34], 90.00th=[ '
'45], 95.00th=[ 53],\n'
' | 99.00th=[ 63], '
'99.50th=[ 67], 99.90th=[ '
'130], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=893891, '
'max=897088, per=100.00%, '
'avg=895900.21, stdev=990.14, '
'samples=14\n'
' iops : min=223472, '
'max=224272, avg=223975.00, '
'stdev=247.65, samples=14\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.01%, 20=0.02%, '
'50=91.17%\n'
' lat (usec) : 100=8.55%, '
'250=0.26%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=18, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=0.7%, 8=99.3%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567643,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=875MiB/s (917MB/s), '
'875MiB/s-875MiB/s '
'(917MB/s-917MB/s), io=6124MiB '
'(6421MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_318.output'),
'state': {'begin': 1666727414.0718408,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7869017124176025,
'end': 1666727414.8587425,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_318.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_318.state')},
'cmd_319': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_319.output'),
'state': {'begin': 1666727416.8832607,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7864069938659668,
'end': 1666727417.6696677,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_319.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_319.state')},
'cmd_32': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_32.output'),
'state': {'begin': 1666726042.852027,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring-reference_1.txt"',
'cwd': 'None',
'elapsed': 11.04966115951538,
'end': 1666726053.901688,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_32.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_32.state')},
'cmd_320': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_320.output'),
'state': {'begin': 1666727417.6715865,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.484341621398926,
'end': 1666727429.1559281,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_320.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_320.state')},
'cmd_321': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734518,\n'
' "timestamp_ms" : '
'1666734518563,\n'
' "time" : "Tue Oct 25 23:48:38 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6416740352,\n'
' "io_kbytes" : 6266348,\n'
' "bw_bytes" : 916546257,\n'
' "bw" : 895064,\n'
' "iops" : 223765.033567,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1566579,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 651,\n'
' "max" : 10229,\n'
' "mean" : 922.635981,\n'
' "stddev" : 74.486330,\n'
' "N" : 1566580\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 13977,\n'
' "max" : 229741,\n'
' "mean" : '
'34515.006953,\n'
' "stddev" : '
'8893.514425,\n'
' "N" : 1566582,\n'
' "percentile" : {\n'
' "1.000000" : 24192,\n'
' "5.000000" : 30080,\n'
' "10.000000" : '
'30336,\n'
' "20.000000" : '
'30592,\n'
' "30.000000" : '
'31104,\n'
' "40.000000" : '
'31872,\n'
' "50.000000" : '
'32384,\n'
' "60.000000" : '
'32640,\n'
' "70.000000" : '
'33024,\n'
' "80.000000" : '
'33536,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'52480,\n'
' "99.000000" : '
'63744,\n'
' "99.500000" : '
'69120,\n'
' "99.900000" : '
'136192,\n'
' "99.950000" : '
'144384,\n'
' "99.990000" : '
'160768\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 14748,\n'
' "max" : 230652,\n'
' "mean" : '
'35437.642907,\n'
' "stddev" : '
'8893.112545,\n'
' "N" : 1566582\n'
' },\n'
' "bw_min" : 892728,\n'
' "bw_max" : 898024,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895313.500000,\n'
' "bw_dev" : 1341.825436,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223182,\n'
' "iops_max" : 224506,\n'
' "iops_mean" : '
'223828.357143,\n'
' "iops_stddev" : '
'335.467895,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 4,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.100000,\n'
' "4" : 0.602140,\n'
' "8" : 99.388540,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.995614,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.048769,\n'
' "50" : 91.253298,\n'
' "100" : 8.411577,\n'
' "250" : 0.286548,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=89020: Tue Oct 25 23:48:38 '
'2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6119MiB/7001msec)\n'
' slat (nsec): min=651, '
'max=10229, avg=922.64, '
'stdev=74.49\n'
' clat (usec): min=13, '
'max=229, avg=34.52, stdev= 8.89\n'
' lat (usec): min=14, '
'max=230, avg=35.44, stdev= 8.89\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 31], 10.00th=[ 31], '
'20.00th=[ 31],\n'
' | 30.00th=[ 32], '
'40.00th=[ 32], 50.00th=[ '
'33], 60.00th=[ 33],\n'
' | 70.00th=[ 34], '
'80.00th=[ 34], 90.00th=[ '
'46], 95.00th=[ 53],\n'
' | 99.00th=[ 64], '
'99.50th=[ 70], 99.90th=[ '
'137], 99.95th=[ 145],\n'
' | 99.99th=[ 161]\n'
' bw ( KiB/s): min=892728, '
'max=898024, per=100.00%, '
'avg=895313.50, stdev=1341.83, '
'samples=14\n'
' iops : min=223182, '
'max=224506, avg=223828.36, '
'stdev=335.47, samples=14\n'
' lat (usec) : 20=0.05%, '
'50=91.25%, 100=8.41%, 250=0.29%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=4, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.1%, '
'4=0.6%, 8=99.4%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1566579,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6119MiB '
'(6417MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_321.output'),
'state': {'begin': 1666727429.158875,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7881069183349609,
'end': 1666727429.946982,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_321.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_321.state')},
'cmd_322': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_322.output'),
'state': {'begin': 1666727431.9935048,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7856662273406982,
'end': 1666727432.779171,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_322.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_322.state')},
'cmd_323': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_323.output'),
'state': {'begin': 1666727432.7809138,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.463064908981323,
'end': 1666727444.2439787,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_323.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_323.state')},
'cmd_324': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734533,\n'
' "timestamp_ms" : '
'1666734533650,\n'
' "time" : "Tue Oct 25 23:48:53 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6419181568,\n'
' "io_kbytes" : 6268732,\n'
' "bw_bytes" : 916894953,\n'
' "bw" : 895405,\n'
' "iops" : 223850.307099,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567176,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 651,\n'
' "max" : 7023,\n'
' "mean" : 994.223934,\n'
' "stddev" : '
'136.149078,\n'
' "N" : 1567176\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 11421,\n'
' "max" : 178565,\n'
' "mean" : '
'34501.146111,\n'
' "stddev" : '
'8517.845352,\n'
' "N" : 1567183,\n'
' "percentile" : {\n'
' "1.000000" : 28544,\n'
' "5.000000" : 30080,\n'
' "10.000000" : '
'30592,\n'
' "20.000000" : '
'31104,\n'
' "30.000000" : '
'31616,\n'
' "40.000000" : '
'31872,\n'
' "50.000000" : '
'31872,\n'
' "60.000000" : '
'32384,\n'
' "70.000000" : '
'32640,\n'
' "80.000000" : '
'33536,\n'
' "90.000000" : '
'44288,\n'
' "95.000000" : '
'52480,\n'
' "99.000000" : '
'63232,\n'
' "99.500000" : '
'66048,\n'
' "99.900000" : '
'130560,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'154624\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12323,\n'
' "max" : 179727,\n'
' "mean" : '
'35495.369933,\n'
' "stddev" : '
'8518.963941,\n'
' "N" : 1567183\n'
' },\n'
' "bw_min" : 892608,\n'
' "bw_max" : 897816,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895630.214286,\n'
' "bw_dev" : 1342.108684,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223152,\n'
' "iops_max" : 224454,\n'
' "iops_mean" : '
'223907.500000,\n'
' "iops_stddev" : '
'335.521351,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 4,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.100000,\n'
' "4" : 0.630050,\n'
' "8" : 99.368801,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.997888,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.013847,\n'
' "50" : 91.109358,\n'
' "100" : 8.605415,\n'
' "250" : 0.271827,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=89070: Tue Oct 25 23:48:53 '
'2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6122MiB/7001msec)\n'
' slat (nsec): min=651, '
'max=7023, avg=994.22, '
'stdev=136.15\n'
' clat (usec): min=11, '
'max=178, avg=34.50, stdev= 8.52\n'
' lat (usec): min=12, '
'max=179, avg=35.50, stdev= 8.52\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 29], '
'5.00th=[ 31], 10.00th=[ 31], '
'20.00th=[ 32],\n'
' | 30.00th=[ 32], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 33],\n'
' | 70.00th=[ 33], '
'80.00th=[ 34], 90.00th=[ '
'45], 95.00th=[ 53],\n'
' | 99.00th=[ 64], '
'99.50th=[ 67], 99.90th=[ '
'131], 99.95th=[ 141],\n'
' | 99.99th=[ 155]\n'
' bw ( KiB/s): min=892608, '
'max=897816, per=100.00%, '
'avg=895630.21, stdev=1342.11, '
'samples=14\n'
' iops : min=223152, '
'max=224454, avg=223907.50, '
'stdev=335.52, samples=14\n'
' lat (usec) : 20=0.01%, '
'50=91.11%, 100=8.61%, 250=0.27%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=4, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.1%, '
'4=0.6%, 8=99.4%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567176,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6122MiB '
'(6419MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_324.output'),
'state': {'begin': 1666727444.2467465,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7881875038146973,
'end': 1666727445.034934,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_324.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_324.state')},
'cmd_325': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_325.output'),
'state': {'begin': 1666727446.0708587,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7909603118896484,
'end': 1666727446.861819,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_325.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_325.state')},
'cmd_326': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_326.output'),
'state': {'begin': 1666727446.8634963,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring-reference_0.txt"',
'cwd': 'None',
'elapsed': 11.056450843811035,
'end': 1666727457.9199471,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_326.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_326.state')},
'cmd_327': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=io_uring, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734547,\n'
' "timestamp_ms" : '
'1666734547398,\n'
' "time" : "Tue Oct 25 23:49:07 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6418763776,\n'
' "io_kbytes" : 6268324,\n'
' "bw_bytes" : 916835277,\n'
' "bw" : 895346,\n'
' "iops" : 223835.594915,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567073,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1222,\n'
' "max" : 43772,\n'
' "mean" : 1732.981963,\n'
' "stddev" : '
'738.776513,\n'
' "N" : 1567074\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 80,\n'
' "max" : 210004,\n'
' "mean" : '
'33654.755580,\n'
' "stddev" : '
'8683.484113,\n'
' "N" : 1567080,\n'
' "percentile" : {\n'
' "1.000000" : 24704,\n'
' "5.000000" : 28544,\n'
' "10.000000" : '
'28800,\n'
' "20.000000" : '
'29568,\n'
' "30.000000" : '
'30592,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31360,\n'
' "60.000000" : '
'31872,\n'
' "70.000000" : '
'32384,\n'
' "80.000000" : '
'33024,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'66048,\n'
' "99.900000" : '
'128512,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 15339,\n'
' "max" : 211277,\n'
' "mean" : '
'35387.737334,\n'
' "stddev" : '
'8632.811278,\n'
' "N" : 1567080\n'
' },\n'
' "bw_min" : 892488,\n'
' "bw_max" : 898184,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895707.214286,\n'
' "bw_dev" : 1505.889371,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223122,\n'
' "iops_max" : 224546,\n'
' "iops_mean" : '
'223926.928571,\n'
' "iops_stddev" : '
'376.422525,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 16.885714,\n'
' "sys_cpu" : 58.414286,\n'
' "ctx" : 669792,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999936,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 0.103697,\n'
' "50" : 92.161693,\n'
' "100" : 7.477826,\n'
' "250" : 0.257040,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2235843,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 70932,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 70932,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=89109: Tue '
'Oct 25 23:49:07 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6121MiB/7001msec)\n'
' slat (nsec): min=1222, '
'max=43772, avg=1732.98, '
'stdev=738.78\n'
' clat (nsec): min=80, '
'max=210004, avg=33654.76, '
'stdev=8683.48\n'
' lat (usec): min=15, '
'max=211, avg=35.39, stdev= 8.63\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 29], 10.00th=[ 29], '
'20.00th=[ 30],\n'
' | 30.00th=[ 31], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 33], '
'80.00th=[ 34], 90.00th=[ '
'46], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 67], 99.90th=[ '
'129], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=892488, '
'max=898184, per=100.00%, '
'avg=895707.21, stdev=1505.89, '
'samples=14\n'
' iops : min=223122, '
'max=224546, avg=223926.93, '
'stdev=376.42, samples=14\n'
' lat (nsec) : 100=0.01%\n'
' lat (usec) : 10=0.01%, '
'20=0.10%, 50=92.16%, 100=7.48%, '
'250=0.26%\n'
' cpu : usr=16.89%, '
'sys=58.41%, ctx=669792, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567073,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6121MiB '
'(6419MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2235843/0, '
'merge=0/0, ticks=70932/0, '
'in_queue=70932, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_327.output'),
'state': {'begin': 1666727457.9227653,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7953412532806396,
'end': 1666727458.7181065,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_327.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_327.state')},
'cmd_328': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_328.output'),
'state': {'begin': 1666727459.7156532,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7870519161224365,
'end': 1666727460.502705,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_328.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_328.state')},
'cmd_329': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_329.output'),
'state': {'begin': 1666727460.504661,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring-reference_1.txt"',
'cwd': 'None',
'elapsed': 11.054227828979492,
'end': 1666727471.558889,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_329.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_329.state')},
'cmd_33': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=io_uring, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733143,\n'
' "timestamp_ms" : '
'1666733143402,\n'
' "time" : "Tue Oct 25 23:25:43 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "io_uring",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1824374784,\n'
' "io_kbytes" : 1781616,\n'
' "bw_bytes" : 260587742,\n'
' "bw" : 254480,\n'
' "iops" : 63620.054278,\n'
' "runtime" : 7001,\n'
' "total_ios" : 445404,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2424,\n'
' "max" : 34645,\n'
' "mean" : 2592.826618,\n'
' "stddev" : 222.766188,\n'
' "N" : 445404\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 240,\n'
' "max" : 122219,\n'
' "mean" : 12494.563899,\n'
' "stddev" : '
'2015.777434,\n'
' "N" : 445404,\n'
' "percentile" : {\n'
' "1.000000" : 11968,\n'
' "5.000000" : 11968,\n'
' "10.000000" : 11968,\n'
' "20.000000" : 12096,\n'
' "30.000000" : 12096,\n'
' "40.000000" : 12224,\n'
' "50.000000" : 12224,\n'
' "60.000000" : 12352,\n'
' "70.000000" : 12352,\n'
' "80.000000" : 12352,\n'
' "90.000000" : 12480,\n'
' "95.000000" : 12480,\n'
' "99.000000" : 22400,\n'
' "99.500000" : 27776,\n'
' "99.900000" : 33024,\n'
' "99.950000" : 49920,\n'
' "99.990000" : 64768\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12434,\n'
' "max" : 124784,\n'
' "mean" : 15087.390517,\n'
' "stddev" : '
'2032.769095,\n'
' "N" : 445404\n'
' },\n'
' "bw_min" : 253272,\n'
' "bw_max" : 255968,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'254578.357143,\n'
' "bw_dev" : 701.625103,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 63318,\n'
' "iops_max" : 63992,\n'
' "iops_mean" : '
'63644.571429,\n'
' "iops_stddev" : '
'175.405778,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 11.200000,\n'
' "sys_cpu" : 36.728571,\n'
' "ctx" : 445388,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 98.600821,\n'
' "50" : 1.339009,\n'
' "100" : 0.052537,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 638282,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 6018,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 6018,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=84810: Tue '
'Oct 25 23:25:43 2022\n'
' read: IOPS=63.6k, BW=249MiB/s '
'(261MB/s)(1740MiB/7001msec)\n'
' slat (nsec): min=2424, '
'max=34645, avg=2592.83, '
'stdev=222.77\n'
' clat (nsec): min=240, '
'max=122219, avg=12494.56, '
'stdev=2015.78\n'
' lat (usec): min=12, max=124, '
'avg=15.09, stdev= 2.03\n'
' clat percentiles (nsec):\n'
' | 1.00th=[11968], '
'5.00th=[11968], 10.00th=[11968], '
'20.00th=[12096],\n'
' | 30.00th=[12096], '
'40.00th=[12224], 50.00th=[12224], '
'60.00th=[12352],\n'
' | 70.00th=[12352], '
'80.00th=[12352], 90.00th=[12480], '
'95.00th=[12480],\n'
' | 99.00th=[22400], '
'99.50th=[27776], 99.90th=[33024], '
'99.95th=[49920],\n'
' | 99.99th=[64768]\n'
' bw ( KiB/s): min=253272, '
'max=255968, per=100.00%, '
'avg=254578.36, stdev=701.63, '
'samples=14\n'
' iops : min=63318, '
'max=63992, avg=63644.57, '
'stdev=175.41, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 10=0.01%, '
'20=98.60%, 50=1.34%, 100=0.05%, '
'250=0.01%\n'
' cpu : usr=11.20%, '
'sys=36.73%, ctx=445388, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=445404,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=249MiB/s (261MB/s), '
'249MiB/s-249MiB/s '
'(261MB/s-261MB/s), io=1740MiB '
'(1824MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=638282/0, '
'merge=0/0, ticks=6018/0, '
'in_queue=6018, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_33.output'),
'state': {'begin': 1666726053.9044824,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7907357215881348,
'end': 1666726054.695218,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_33.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_33.state')},
'cmd_330': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=io_uring, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734561,\n'
' "timestamp_ms" : '
'1666734561035,\n'
' "time" : "Tue Oct 25 23:49:21 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6418571264,\n'
' "io_kbytes" : 6268136,\n'
' "bw_bytes" : 916807779,\n'
' "bw" : 895320,\n'
' "iops" : 223828.881588,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567026,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1212,\n'
' "max" : 36078,\n'
' "mean" : 1714.383513,\n'
' "stddev" : '
'722.179076,\n'
' "N" : 1567027\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 12804,\n'
' "max" : 206107,\n'
' "mean" : '
'33675.974473,\n'
' "stddev" : '
'8691.726209,\n'
' "N" : 1567033,\n'
' "percentile" : {\n'
' "1.000000" : 24960,\n'
' "5.000000" : 28544,\n'
' "10.000000" : '
'28800,\n'
' "20.000000" : '
'29824,\n'
' "30.000000" : '
'30592,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31360,\n'
' "60.000000" : '
'31616,\n'
' "70.000000" : '
'32384,\n'
' "80.000000" : '
'33024,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'128512,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 15399,\n'
' "max" : 207379,\n'
' "mean" : '
'35390.359061,\n'
' "stddev" : '
'8641.981671,\n'
' "N" : 1567033\n'
' },\n'
' "bw_min" : 893378,\n'
' "bw_max" : 897136,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895682.857143,\n'
' "bw_dev" : 1185.473801,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223344,\n'
' "iops_max" : 224284,\n'
' "iops_mean" : '
'223920.642857,\n'
' "iops_stddev" : '
'296.455888,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 16.600000,\n'
' "sys_cpu" : 58.057143,\n'
' "ctx" : 693034,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999936,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.086406,\n'
' "50" : 92.191195,\n'
' "100" : 7.458842,\n'
' "250" : 0.264003,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2235997,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 71144,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 71144,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=89147: Tue '
'Oct 25 23:49:21 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6121MiB/7001msec)\n'
' slat (nsec): min=1212, '
'max=36078, avg=1714.38, '
'stdev=722.18\n'
' clat (usec): min=12, '
'max=206, avg=33.68, stdev= 8.69\n'
' lat (usec): min=15, '
'max=207, avg=35.39, stdev= 8.64\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 29], 10.00th=[ 29], '
'20.00th=[ 30],\n'
' | 30.00th=[ 31], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 33], '
'80.00th=[ 34], 90.00th=[ '
'46], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 68], 99.90th=[ '
'129], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=893378, '
'max=897136, per=100.00%, '
'avg=895682.86, stdev=1185.47, '
'samples=14\n'
' iops : min=223344, '
'max=224284, avg=223920.64, '
'stdev=296.46, samples=14\n'
' lat (usec) : 20=0.09%, '
'50=92.19%, 100=7.46%, 250=0.26%\n'
' cpu : usr=16.60%, '
'sys=58.06%, ctx=693034, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567026,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6121MiB '
'(6419MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2235997/0, '
'merge=0/0, ticks=71144/0, '
'in_queue=71144, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_330.output'),
'state': {'begin': 1666727471.5615969,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7940971851348877,
'end': 1666727472.355694,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_330.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_330.state')},
'cmd_331': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_331.output'),
'state': {'begin': 1666727473.3489244,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7900288105010986,
'end': 1666727474.1389532,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_331.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_331.state')},
'cmd_332': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_332.output'),
'state': {'begin': 1666727474.1414618,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring-reference_2.txt"',
'cwd': 'None',
'elapsed': 11.046249628067017,
'end': 1666727485.1877115,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_332.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_332.state')},
'cmd_333': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=io_uring, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734574,\n'
' "timestamp_ms" : '
'1666734574665,\n'
' "time" : "Tue Oct 25 23:49:34 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6419468288,\n'
' "io_kbytes" : 6269012,\n'
' "bw_bytes" : 916935907,\n'
' "bw" : 895445,\n'
' "iops" : 223860.162834,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567245,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1232,\n'
' "max" : 53460,\n'
' "mean" : 1707.207037,\n'
' "stddev" : '
'717.038812,\n'
' "N" : 1567246\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 7434,\n'
' "max" : 196628,\n'
' "mean" : '
'33672.910071,\n'
' "stddev" : '
'8730.705101,\n'
' "N" : 1567252,\n'
' "percentile" : {\n'
' "1.000000" : 25216,\n'
' "5.000000" : 28544,\n'
' "10.000000" : '
'28800,\n'
' "20.000000" : '
'29312,\n'
' "30.000000" : '
'30592,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31360,\n'
' "60.000000" : '
'31872,\n'
' "70.000000" : '
'32384,\n'
' "80.000000" : '
'33024,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'129536,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 15579,\n'
' "max" : 198051,\n'
' "mean" : '
'35380.117355,\n'
' "stddev" : '
'8681.842147,\n'
' "N" : 1567252\n'
' },\n'
' "bw_min" : 892529,\n'
' "bw_max" : 897192,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895805.071429,\n'
' "bw_dev" : 1222.038050,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223132,\n'
' "iops_max" : 224298,\n'
' "iops_mean" : '
'223951.357143,\n'
' "iops_stddev" : '
'305.643233,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.300000,\n'
' "sys_cpu" : 57.928571,\n'
' "ctx" : 667803,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999936,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 0.066550,\n'
' "50" : 92.209706,\n'
' "100" : 7.450845,\n'
' "250" : 0.273154,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2236081,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 71107,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 71107,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=89185: Tue '
'Oct 25 23:49:34 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6122MiB/7001msec)\n'
' slat (nsec): min=1232, '
'max=53460, avg=1707.21, '
'stdev=717.04\n'
' clat (usec): min=7, max=196, '
'avg=33.67, stdev= 8.73\n'
' lat (usec): min=15, '
'max=198, avg=35.38, stdev= 8.68\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 26], '
'5.00th=[ 29], 10.00th=[ 29], '
'20.00th=[ 30],\n'
' | 30.00th=[ 31], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 33], '
'80.00th=[ 34], 90.00th=[ '
'46], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 68], 99.90th=[ '
'130], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=892529, '
'max=897192, per=100.00%, '
'avg=895805.07, stdev=1222.04, '
'samples=14\n'
' iops : min=223132, '
'max=224298, avg=223951.36, '
'stdev=305.64, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=0.07%, 50=92.21%, 100=7.45%, '
'250=0.27%\n'
' cpu : usr=17.30%, '
'sys=57.93%, ctx=667803, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567245,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6122MiB '
'(6419MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2236081/0, '
'merge=0/0, ticks=71107/0, '
'in_queue=71107, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_333.output'),
'state': {'begin': 1666727485.1905935,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7888400554656982,
'end': 1666727485.9794335,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_333.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_333.state')},
'cmd_334': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_334.output'),
'state': {'begin': 1666727488.0161452,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.7870473861694336,
'end': 1666727488.8031926,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_334.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_334.state')},
'cmd_335': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_335.output'),
'state': {'begin': 1666727488.8052003,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt"',
'cwd': 'None',
'elapsed': 11.470963478088379,
'end': 1666727500.2761638,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_335.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_335.state')},
'cmd_336': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734589,\n'
' "timestamp_ms" : '
'1666734589681,\n'
' "time" : "Tue Oct 25 23:49:49 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6419386368,\n'
' "io_kbytes" : 6268932,\n'
' "bw_bytes" : 916924206,\n'
' "bw" : 895433,\n'
' "iops" : 223857.448936,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567226,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 17783,\n'
' "mean" : 177.147500,\n'
' "stddev" : 78.228730,\n'
' "N" : 1567226\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 13115,\n'
' "max" : 1356474,\n'
' "mean" : '
'35300.071299,\n'
' "stddev" : '
'8917.372350,\n'
' "N" : 1567233,\n'
' "percentile" : {\n'
' "1.000000" : 29568,\n'
' "5.000000" : 32384,\n'
' "10.000000" : '
'32384,\n'
' "20.000000" : '
'32640,\n'
' "30.000000" : '
'32640,\n'
' "40.000000" : '
'32640,\n'
' "50.000000" : '
'32640,\n'
' "60.000000" : '
'32640,\n'
' "70.000000" : '
'33024,\n'
' "80.000000" : '
'33024,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'53504,\n'
' "99.000000" : '
'63744,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'130560,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 13395,\n'
' "max" : 1356644,\n'
' "mean" : '
'35477.218807,\n'
' "stddev" : '
'8916.741132,\n'
' "N" : 1567233\n'
' },\n'
' "bw_min" : 893360,\n'
' "bw_max" : 897640,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895644.571429,\n'
' "bw_dev" : 1100.192830,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223340,\n'
' "iops_max" : 224410,\n'
' "iops_mean" : '
'223911.142857,\n'
' "iops_stddev" : '
'275.048208,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 39.585714,\n'
' "sys_cpu" : 60.385714,\n'
' "ctx" : 40,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.100000,\n'
' "4" : 1.291773,\n'
' "8" : 98.704335,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.996949,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.010000,\n'
' "50" : 90.676265,\n'
' "100" : 9.055108,\n'
' "250" : 0.262374,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=89233: Tue '
'Oct 25 23:49:49 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6122MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=17783, avg=177.15, '
'stdev=78.23\n'
' clat (usec): min=13, '
'max=1356, avg=35.30, stdev= '
'8.92\n'
' lat (usec): min=13, '
'max=1356, avg=35.48, stdev= '
'8.92\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 30], '
'5.00th=[ 33], 10.00th=[ 33], '
'20.00th=[ 33],\n'
' | 30.00th=[ 33], '
'40.00th=[ 33], 50.00th=[ '
'33], 60.00th=[ 33],\n'
' | 70.00th=[ 34], '
'80.00th=[ 34], 90.00th=[ '
'46], 95.00th=[ 54],\n'
' | 99.00th=[ 64], '
'99.50th=[ 68], 99.90th=[ '
'131], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=893360, '
'max=897640, per=100.00%, '
'avg=895644.57, stdev=1100.19, '
'samples=14\n'
' iops : min=223340, '
'max=224410, avg=223911.14, '
'stdev=275.05, samples=14\n'
' lat (usec) : 20=0.01%, '
'50=90.68%, 100=9.06%, 250=0.26%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=39.59%, '
'sys=60.39%, ctx=40, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=1.3%, 8=98.7%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567226,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6122MiB '
'(6419MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_336.output'),
'state': {'begin': 1666727500.2788785,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.8842198848724365,
'end': 1666727501.1630983,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_336.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_336.state')},
'cmd_337': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_337.output'),
'state': {'begin': 1666727503.4680705,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.8344917297363281,
'end': 1666727504.3025622,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_337.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_337.state')},
'cmd_338': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_338.output'),
'state': {'begin': 1666727504.3044047,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt"',
'cwd': 'None',
'elapsed': 11.516225814819336,
'end': 1666727515.8206306,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_338.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_338.state')},
'cmd_339': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734605,\n'
' "timestamp_ms" : '
'1666734605226,\n'
' "time" : "Tue Oct 25 23:50:05 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6418595840,\n'
' "io_kbytes" : 6268160,\n'
' "bw_bytes" : 916811289,\n'
' "bw" : 895323,\n'
' "iops" : 223829.881446,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567033,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 120,\n'
' "max" : 18484,\n'
' "mean" : 180.659000,\n'
' "stddev" : 80.296904,\n'
' "N" : 1567033\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 13105,\n'
' "max" : 191649,\n'
' "mean" : '
'35285.179822,\n'
' "stddev" : '
'8502.394509,\n'
' "N" : 1567040,\n'
' "percentile" : {\n'
' "1.000000" : 29824,\n'
' "5.000000" : 32384,\n'
' "10.000000" : '
'32384,\n'
' "20.000000" : '
'32640,\n'
' "30.000000" : '
'32640,\n'
' "40.000000" : '
'32640,\n'
' "50.000000" : '
'32640,\n'
' "60.000000" : '
'32640,\n'
' "70.000000" : '
'33024,\n'
' "80.000000" : '
'33024,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'53504,\n'
' "99.000000" : '
'63744,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'132096,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 13355,\n'
' "max" : 191830,\n'
' "mean" : '
'35465.838813,\n'
' "stddev" : '
'8501.391034,\n'
' "N" : 1567040\n'
' },\n'
' "bw_min" : 891984,\n'
' "bw_max" : 898280,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895534.142857,\n'
' "bw_dev" : 1537.441324,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 222996,\n'
' "iops_max" : 224570,\n'
' "iops_mean" : '
'223883.500000,\n'
' "iops_stddev" : '
'384.365761,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 42.957143,\n'
' "sys_cpu" : 57.014286,\n'
' "ctx" : 11,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.100000,\n'
' "4" : 1.244645,\n'
' "8" : 98.750441,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.996629,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.010000,\n'
' "50" : 90.670713,\n'
' "100" : 9.049012,\n'
' "250" : 0.275425,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=89282: Tue '
'Oct 25 23:50:05 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6121MiB/7001msec)\n'
' slat (nsec): min=120, '
'max=18484, avg=180.66, '
'stdev=80.30\n'
' clat (usec): min=13, '
'max=191, avg=35.29, stdev= 8.50\n'
' lat (usec): min=13, '
'max=191, avg=35.47, stdev= 8.50\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 30], '
'5.00th=[ 33], 10.00th=[ 33], '
'20.00th=[ 33],\n'
' | 30.00th=[ 33], '
'40.00th=[ 33], 50.00th=[ '
'33], 60.00th=[ 33],\n'
' | 70.00th=[ 34], '
'80.00th=[ 34], 90.00th=[ '
'46], 95.00th=[ 54],\n'
' | 99.00th=[ 64], '
'99.50th=[ 68], 99.90th=[ '
'133], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=891984, '
'max=898280, per=100.00%, '
'avg=895534.14, stdev=1537.44, '
'samples=14\n'
' iops : min=222996, '
'max=224570, avg=223883.50, '
'stdev=384.37, samples=14\n'
' lat (usec) : 20=0.01%, '
'50=90.67%, 100=9.05%, 250=0.28%\n'
' cpu : usr=42.96%, '
'sys=57.01%, ctx=11, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=1.2%, 8=98.8%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567033,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6121MiB '
'(6419MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_339.output'),
'state': {'begin': 1666727515.8234217,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.8254802227020264,
'end': 1666727516.648902,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_339.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_339.state')},
'cmd_34': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_34.output'),
'state': {'begin': 1666726055.6835701,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7879164218902588,
'end': 1666726056.4714866,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_34.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_34.state')},
'cmd_340': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_340.output'),
'state': {'begin': 1666727518.7496185,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.813143253326416,
'end': 1666727519.5627618,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_340.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_340.state')},
'cmd_341': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_341.output'),
'state': {'begin': 1666727519.5640981,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt"',
'cwd': 'None',
'elapsed': 11.459917545318604,
'end': 1666727531.0240157,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_341.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_341.state')},
'cmd_342': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734620,\n'
' "timestamp_ms" : '
'1666734620429,\n'
' "time" : "Tue Oct 25 23:50:20 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6415040512,\n'
' "io_kbytes" : 6264688,\n'
' "bw_bytes" : 916303458,\n'
' "bw" : 894827,\n'
' "iops" : 223705.899157,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1566165,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 41308,\n'
' "mean" : 199.295783,\n'
' "stddev" : '
'128.404646,\n'
' "N" : 1566165\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 12804,\n'
' "max" : 177943,\n'
' "mean" : '
'35310.666389,\n'
' "stddev" : '
'8531.519283,\n'
' "N" : 1566172,\n'
' "percentile" : {\n'
' "1.000000" : 29568,\n'
' "5.000000" : 32384,\n'
' "10.000000" : '
'32384,\n'
' "20.000000" : '
'32640,\n'
' "30.000000" : '
'32640,\n'
' "40.000000" : '
'32640,\n'
' "50.000000" : '
'32640,\n'
' "60.000000" : '
'32640,\n'
' "70.000000" : '
'33024,\n'
' "80.000000" : '
'33024,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'53504,\n'
' "99.000000" : '
'63744,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'132096,\n'
' "99.950000" : '
'142336,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 13055,\n'
' "max" : 178154,\n'
' "mean" : '
'35509.962140,\n'
' "stddev" : '
'8530.220824,\n'
' "N" : 1566172\n'
' },\n'
' "bw_min" : 893232,\n'
' "bw_max" : 895976,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895036.428571,\n'
' "bw_dev" : 819.653277,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223308,\n'
' "iops_max" : 223994,\n'
' "iops_mean" : '
'223759.214286,\n'
' "iops_stddev" : '
'205.000630,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 39.642857,\n'
' "sys_cpu" : 60.328571,\n'
' "ctx" : 19,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.100000,\n'
' "4" : 1.353178,\n'
' "8" : 98.618983,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.991192,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.011110,\n'
' "50" : 90.645749,\n'
' "100" : 9.069670,\n'
' "250" : 0.273917,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=89331: Tue '
'Oct 25 23:50:20 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(916MB/s)(6118MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=41308, avg=199.30, '
'stdev=128.40\n'
' clat (usec): min=12, '
'max=177, avg=35.31, stdev= 8.53\n'
' lat (usec): min=13, '
'max=178, avg=35.51, stdev= 8.53\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 30], '
'5.00th=[ 33], 10.00th=[ 33], '
'20.00th=[ 33],\n'
' | 30.00th=[ 33], '
'40.00th=[ 33], 50.00th=[ '
'33], 60.00th=[ 33],\n'
' | 70.00th=[ 34], '
'80.00th=[ 34], 90.00th=[ '
'46], 95.00th=[ 54],\n'
' | 99.00th=[ 64], '
'99.50th=[ 68], 99.90th=[ '
'133], 99.95th=[ 143],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=893232, '
'max=895976, per=100.00%, '
'avg=895036.43, stdev=819.65, '
'samples=14\n'
' iops : min=223308, '
'max=223994, avg=223759.21, '
'stdev=205.00, samples=14\n'
' lat (usec) : 20=0.01%, '
'50=90.65%, 100=9.07%, 250=0.27%\n'
' cpu : usr=39.64%, '
'sys=60.33%, ctx=19, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=1.4%, 8=98.6%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1566165,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (916MB/s), '
'874MiB/s-874MiB/s '
'(916MB/s-916MB/s), io=6118MiB '
'(6415MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_342.output'),
'state': {'begin': 1666727531.026178,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.7857508659362793,
'end': 1666727531.8119287,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_342.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_342.state')},
'cmd_343': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_343.output'),
'state': {'begin': 1666727533.9081874,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7813472747802734,
'end': 1666727534.6895347,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_343.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_343.state')},
'cmd_344': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_344.output'),
'state': {'begin': 1666727534.69146,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.462684392929077,
'end': 1666727546.1541443,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_344.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_344.state')},
'cmd_345': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734635,\n'
' "timestamp_ms" : '
'1666734635553,\n'
' "time" : "Tue Oct 25 23:50:35 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6416384000,\n'
' "io_kbytes" : 6266000,\n'
' "bw_bytes" : 916495357,\n'
' "bw" : 895014,\n'
' "iops" : 223752.749607,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1566493,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1482,\n'
' "max" : 67687,\n'
' "mean" : 1754.159928,\n'
' "stddev" : '
'246.058703,\n'
' "N" : 1566493\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 15098,\n'
' "max" : 202079,\n'
' "mean" : '
'33741.697037,\n'
' "stddev" : '
'8649.734313,\n'
' "N" : 1566500,\n'
' "percentile" : {\n'
' "1.000000" : 24448,\n'
' "5.000000" : 30848,\n'
' "10.000000" : '
'30848,\n'
' "20.000000" : '
'30848,\n'
' "30.000000" : '
'31104,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31104,\n'
' "60.000000" : '
'31104,\n'
' "70.000000" : '
'31360,\n'
' "80.000000" : '
'31360,\n'
' "90.000000" : '
'43776,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62208,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'132096,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 17352,\n'
' "max" : 203803,\n'
' "mean" : '
'35495.856783,\n'
' "stddev" : '
'8639.780490,\n'
' "N" : 1566500\n'
' },\n'
' "bw_min" : 894056,\n'
' "bw_max" : 897080,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895219.785714,\n'
' "bw_dev" : 929.395517,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223514,\n'
' "iops_max" : 224270,\n'
' "iops_mean" : '
'223804.785714,\n'
' "iops_stddev" : '
'232.359198,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 30.385714,\n'
' "sys_cpu" : 69.585714,\n'
' "ctx" : 14,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.026684,\n'
' "50" : 91.094183,\n'
' "100" : 8.585867,\n'
' "250" : 0.293713,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=89380: Tue '
'Oct 25 23:50:35 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(916MB/s)(6119MiB/7001msec)\n'
' slat (nsec): min=1482, '
'max=67687, avg=1754.16, '
'stdev=246.06\n'
' clat (usec): min=15, '
'max=202, avg=33.74, stdev= 8.65\n'
' lat (usec): min=17, '
'max=203, avg=35.50, stdev= 8.64\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 31], 10.00th=[ 31], '
'20.00th=[ 31],\n'
' | 30.00th=[ 32], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 32], '
'80.00th=[ 32], 90.00th=[ '
'44], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 68], 99.90th=[ '
'133], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=894056, '
'max=897080, per=100.00%, '
'avg=895219.79, stdev=929.40, '
'samples=14\n'
' iops : min=223514, '
'max=224270, avg=223804.79, '
'stdev=232.36, samples=14\n'
' lat (usec) : 20=0.03%, '
'50=91.09%, 100=8.59%, 250=0.29%\n'
' cpu : usr=30.39%, '
'sys=69.59%, ctx=14, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1566493,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (916MB/s), '
'874MiB/s-874MiB/s '
'(916MB/s-916MB/s), io=6119MiB '
'(6416MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_345.output'),
'state': {'begin': 1666727546.1562865,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8221015930175781,
'end': 1666727546.978388,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_345.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_345.state')},
'cmd_346': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_346.output'),
'state': {'begin': 1666727549.0630267,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7824573516845703,
'end': 1666727549.845484,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_346.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_346.state')},
'cmd_347': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_347.output'),
'state': {'begin': 1666727549.8478909,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.456165552139282,
'end': 1666727561.3040564,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_347.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_347.state')},
'cmd_348': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734650,\n'
' "timestamp_ms" : '
'1666734650706,\n'
' "time" : "Tue Oct 25 23:50:50 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6418513920,\n'
' "io_kbytes" : 6268080,\n'
' "bw_bytes" : 916799588,\n'
' "bw" : 895312,\n'
' "iops" : 223827.024711,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567013,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1473,\n'
' "max" : 25368,\n'
' "mean" : 1749.800455,\n'
' "stddev" : '
'234.769642,\n'
' "N" : 1567013\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 12604,\n'
' "max" : 1083192,\n'
' "mean" : '
'33745.384779,\n'
' "stddev" : '
'8890.164459,\n'
' "N" : 1567020,\n'
' "percentile" : {\n'
' "1.000000" : 24448,\n'
' "5.000000" : 30848,\n'
' "10.000000" : '
'30848,\n'
' "20.000000" : '
'30848,\n'
' "30.000000" : '
'31104,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31104,\n'
' "60.000000" : '
'31104,\n'
' "70.000000" : '
'31360,\n'
' "80.000000" : '
'31360,\n'
' "90.000000" : '
'43776,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62208,\n'
' "99.500000" : '
'66048,\n'
' "99.900000" : '
'130560,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 14317,\n'
' "max" : 1084925,\n'
' "mean" : '
'35495.185114,\n'
' "stddev" : '
'8880.807151,\n'
' "N" : 1567020\n'
' },\n'
' "bw_min" : 893456,\n'
' "bw_max" : 896816,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895527.357143,\n'
' "bw_dev" : 964.990044,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223364,\n'
' "iops_max" : 224204,\n'
' "iops_mean" : '
'223881.785714,\n'
' "iops_stddev" : '
'241.228962,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 25.757143,\n'
' "sys_cpu" : 74.214286,\n'
' "ctx" : 42,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.046075,\n'
' "50" : 91.074356,\n'
' "100" : 8.607331,\n'
' "250" : 0.272174,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=89429: Tue '
'Oct 25 23:50:50 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6121MiB/7001msec)\n'
' slat (nsec): min=1473, '
'max=25368, avg=1749.80, '
'stdev=234.77\n'
' clat (usec): min=12, '
'max=1083, avg=33.75, stdev= '
'8.89\n'
' lat (usec): min=14, '
'max=1084, avg=35.50, stdev= '
'8.88\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 31], 10.00th=[ 31], '
'20.00th=[ 31],\n'
' | 30.00th=[ 32], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 32], '
'80.00th=[ 32], 90.00th=[ '
'44], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 67], 99.90th=[ '
'131], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=893456, '
'max=896816, per=100.00%, '
'avg=895527.36, stdev=964.99, '
'samples=14\n'
' iops : min=223364, '
'max=224204, avg=223881.79, '
'stdev=241.23, samples=14\n'
' lat (usec) : 20=0.05%, '
'50=91.07%, 100=8.61%, 250=0.27%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=25.76%, '
'sys=74.21%, ctx=42, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567013,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6121MiB '
'(6419MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_348.output'),
'state': {'begin': 1666727561.3071404,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7886478900909424,
'end': 1666727562.0957882,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_348.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_348.state')},
'cmd_349': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_349.output'),
'state': {'begin': 1666727564.1502848,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7863068580627441,
'end': 1666727564.9365916,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_349.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_349.state')},
'cmd_35': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_35.output'),
'state': {'begin': 1666726056.473827,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring-reference_2.txt"',
'cwd': 'None',
'elapsed': 11.052614212036133,
'end': 1666726067.526441,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_35.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_35.state')},
'cmd_350': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_350.output'),
'state': {'begin': 1666727564.9434702,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.563092470169067,
'end': 1666727576.5065627,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_350.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_350.state')},
'cmd_351': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734665,\n'
' "timestamp_ms" : '
'1666734665909,\n'
' "time" : "Tue Oct 25 23:51:05 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6418640896,\n'
' "io_kbytes" : 6268204,\n'
' "bw_bytes" : 916817725,\n'
' "bw" : 895329,\n'
' "iops" : 223831.452650,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567044,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1472,\n'
' "max" : 25949,\n'
' "mean" : 1754.159593,\n'
' "stddev" : '
'224.643680,\n'
' "N" : 1567044\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 15079,\n'
' "max" : 220213,\n'
' "mean" : '
'33737.245896,\n'
' "stddev" : '
'8603.637014,\n'
' "N" : 1567051,\n'
' "percentile" : {\n'
' "1.000000" : 24704,\n'
' "5.000000" : 30848,\n'
' "10.000000" : '
'30848,\n'
' "20.000000" : '
'30848,\n'
' "30.000000" : '
'31104,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31104,\n'
' "60.000000" : '
'31104,\n'
' "70.000000" : '
'31360,\n'
' "80.000000" : '
'31360,\n'
' "90.000000" : '
'43776,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62208,\n'
' "99.500000" : '
'66048,\n'
' "99.900000" : '
'132096,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 17322,\n'
' "max" : 221956,\n'
' "mean" : '
'35491.405287,\n'
' "stddev" : '
'8595.079541,\n'
' "N" : 1567051\n'
' },\n'
' "bw_min" : 892688,\n'
' "bw_max" : 897683,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895548.785714,\n'
' "bw_dev" : 1287.687264,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223172,\n'
' "iops_max" : 224420,\n'
' "iops_mean" : '
'223887.142857,\n'
' "iops_stddev" : '
'321.708625,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 29.585714,\n'
' "sys_cpu" : 70.385714,\n'
' "ctx" : 13,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.035098,\n'
' "50" : 91.088827,\n'
' "100" : 8.597908,\n'
' "250" : 0.278614,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=89479: Tue '
'Oct 25 23:51:05 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6121MiB/7001msec)\n'
' slat (nsec): min=1472, '
'max=25949, avg=1754.16, '
'stdev=224.64\n'
' clat (usec): min=15, '
'max=220, avg=33.74, stdev= 8.60\n'
' lat (usec): min=17, '
'max=221, avg=35.49, stdev= 8.60\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 31], 10.00th=[ 31], '
'20.00th=[ 31],\n'
' | 30.00th=[ 32], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 32], '
'80.00th=[ 32], 90.00th=[ '
'44], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 67], 99.90th=[ '
'133], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=892688, '
'max=897683, per=100.00%, '
'avg=895548.79, stdev=1287.69, '
'samples=14\n'
' iops : min=223172, '
'max=224420, avg=223887.14, '
'stdev=321.71, samples=14\n'
' lat (usec) : 20=0.04%, '
'50=91.09%, 100=8.60%, 250=0.28%\n'
' cpu : usr=29.59%, '
'sys=70.39%, ctx=13, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567044,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6121MiB '
'(6419MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_351.output'),
'state': {'begin': 1666727576.5091283,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7935681343078613,
'end': 1666727577.3026965,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_351.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_351.state')},
'cmd_352': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_352.output'),
'state': {'begin': 1666727578.3296173,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8199388980865479,
'end': 1666727579.1495562,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_352.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_352.state')},
'cmd_353': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_353.output'),
'state': {'begin': 1666727579.1509423,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring-xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.05217170715332,
'end': 1666727590.203114,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_353.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_353.state')},
'cmd_354': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734679,\n'
' "timestamp_ms" : '
'1666734679679,\n'
' "time" : "Tue Oct 25 23:51:19 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6417080320,\n'
' "io_kbytes" : 6266680,\n'
' "bw_bytes" : 916594817,\n'
' "bw" : 895112,\n'
' "iops" : 223777.031853,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1566663,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1312,\n'
' "max" : 178064,\n'
' "mean" : 2022.793881,\n'
' "stddev" : '
'860.278382,\n'
' "N" : 1566663\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5811,\n'
' "max" : 218590,\n'
' "mean" : '
'33429.997546,\n'
' "stddev" : '
'8920.942316,\n'
' "N" : 1566670,\n'
' "percentile" : {\n'
' "1.000000" : 23936,\n'
' "5.000000" : 28032,\n'
' "10.000000" : '
'28032,\n'
' "20.000000" : '
'28544,\n'
' "30.000000" : '
'30336,\n'
' "40.000000" : '
'30848,\n'
' "50.000000" : '
'31872,\n'
' "60.000000" : '
'32128,\n'
' "70.000000" : '
'32384,\n'
' "80.000000" : '
'32640,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62208,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'132096,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 15940,\n'
' "max" : 221325,\n'
' "mean" : '
'35452.790505,\n'
' "stddev" : '
'8811.650507,\n'
' "N" : 1566670\n'
' },\n'
' "bw_min" : 894184,\n'
' "bw_max" : 897432,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895345.000000,\n'
' "bw_dev" : 952.534675,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223546,\n'
' "iops_max" : 224358,\n'
' "iops_mean" : '
'223836.214286,\n'
' "iops_stddev" : '
'238.146748,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 16.414286,\n'
' "sys_cpu" : 62.785714,\n'
' "ctx" : 539835,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 0.228958,\n'
' "50" : 92.130024,\n'
' "100" : 7.357294,\n'
' "250" : 0.284107,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=89518: Tue '
'Oct 25 23:51:19 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6120MiB/7001msec)\n'
' slat (nsec): min=1312, '
'max=178064, avg=2022.79, '
'stdev=860.28\n'
' clat (usec): min=5, max=218, '
'avg=33.43, stdev= 8.92\n'
' lat (usec): min=15, '
'max=221, avg=35.45, stdev= 8.81\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 24], '
'5.00th=[ 29], 10.00th=[ 29], '
'20.00th=[ 29],\n'
' | 30.00th=[ 31], '
'40.00th=[ 31], 50.00th=[ '
'32], 60.00th=[ 33],\n'
' | 70.00th=[ 33], '
'80.00th=[ 33], 90.00th=[ '
'46], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 68], 99.90th=[ '
'133], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=894184, '
'max=897432, per=100.00%, '
'avg=895345.00, stdev=952.53, '
'samples=14\n'
' iops : min=223546, '
'max=224358, avg=223836.21, '
'stdev=238.15, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=0.23%, 50=92.13%, 100=7.36%, '
'250=0.28%\n'
' cpu : usr=16.41%, '
'sys=62.79%, ctx=539835, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1566663,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6120MiB '
'(6417MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_354.output'),
'state': {'begin': 1666727590.2101974,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7959024906158447,
'end': 1666727591.0061,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_354.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_354.state')},
'cmd_355': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_355.output'),
'state': {'begin': 1666727591.9957767,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7794620990753174,
'end': 1666727592.7752388,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_355.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_355.state')},
'cmd_356': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_356.output'),
'state': {'begin': 1666727592.7774534,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring-xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.05287218093872,
'end': 1666727603.8303256,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_356.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_356.state')},
'cmd_357': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734693,\n'
' "timestamp_ms" : '
'1666734693305,\n'
' "time" : "Tue Oct 25 23:51:33 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6416519168,\n'
' "io_kbytes" : 6266132,\n'
' "bw_bytes" : 916514664,\n'
' "bw" : 895033,\n'
' "iops" : 223757.320383,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1566525,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1283,\n'
' "max" : 37260,\n'
' "mean" : 2104.457766,\n'
' "stddev" : '
'885.717940,\n'
' "N" : 1566526\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 220,\n'
' "max" : 267692,\n'
' "mean" : '
'33332.316245,\n'
' "stddev" : '
'8926.514508,\n'
' "N" : 1566532,\n'
' "percentile" : {\n'
' "1.000000" : 24192,\n'
' "5.000000" : 28032,\n'
' "10.000000" : '
'28032,\n'
' "20.000000" : '
'28288,\n'
' "30.000000" : '
'29568,\n'
' "40.000000" : '
'30848,\n'
' "50.000000" : '
'31872,\n'
' "60.000000" : '
'32128,\n'
' "70.000000" : '
'32384,\n'
' "80.000000" : '
'32640,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62208,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'132096,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 15189,\n'
' "max" : 269315,\n'
' "mean" : '
'35436.773715,\n'
' "stddev" : '
'8807.724468,\n'
' "N" : 1566532\n'
' },\n'
' "bw_min" : 892808,\n'
' "bw_max" : 897400,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895258.142857,\n'
' "bw_dev" : 1198.668098,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223202,\n'
' "iops_max" : 224350,\n'
' "iops_mean" : '
'223814.500000,\n'
' "iops_stddev" : '
'299.664107,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.414286,\n'
' "sys_cpu" : 64.100000,\n'
' "ctx" : 529590,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 0.256364,\n'
' "50" : 92.233191,\n'
' "100" : 7.231356,\n'
' "250" : 0.279089,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=89557: Tue '
'Oct 25 23:51:33 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6119MiB/7001msec)\n'
' slat (nsec): min=1283, '
'max=37260, avg=2104.46, '
'stdev=885.72\n'
' clat (nsec): min=220, '
'max=267692, avg=33332.32, '
'stdev=8926.51\n'
' lat (usec): min=15, '
'max=269, avg=35.44, stdev= 8.81\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 29], 10.00th=[ 29], '
'20.00th=[ 29],\n'
' | 30.00th=[ 30], '
'40.00th=[ 31], 50.00th=[ '
'32], 60.00th=[ 33],\n'
' | 70.00th=[ 33], '
'80.00th=[ 33], 90.00th=[ '
'46], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 68], 99.90th=[ '
'133], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=892808, '
'max=897400, per=100.00%, '
'avg=895258.14, stdev=1198.67, '
'samples=14\n'
' iops : min=223202, '
'max=224350, avg=223814.50, '
'stdev=299.66, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=0.26%, 50=92.23%, '
'100=7.23%\n'
' lat (usec) : 250=0.28%, '
'500=0.01%\n'
' cpu : usr=15.41%, '
'sys=64.10%, ctx=529590, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1566525,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6119MiB '
'(6417MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_357.output'),
'state': {'begin': 1666727603.8315434,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.783583402633667,
'end': 1666727604.6151268,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_357.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_357.state')},
'cmd_358': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_io_uring-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_358.output'),
'state': {'begin': 1666727605.6035314,
'cmd': 'rm '
'/tmp/fio-output_4K_8_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7908978462219238,
'end': 1666727606.3944292,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_358.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_358.state')},
'cmd_359': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_359.output'),
'state': {'begin': 1666727606.3975043,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_io_uring-xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.065941095352173,
'end': 1666727617.4634454,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_359.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_359.state')},
'cmd_36': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=io_uring, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733157,\n'
' "timestamp_ms" : '
'1666733157028,\n'
' "time" : "Tue Oct 25 23:25:57 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "io_uring",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1804107776,\n'
' "io_kbytes" : 1761824,\n'
' "bw_bytes" : 257729682,\n'
' "bw" : 251689,\n'
' "iops" : 62922.285714,\n'
' "runtime" : 7000,\n'
' "total_ios" : 440456,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2465,\n'
' "max" : 34736,\n'
' "mean" : 2646.838867,\n'
' "stddev" : 213.387163,\n'
' "N" : 440456\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 240,\n'
' "max" : 105898,\n'
' "mean" : 12567.897549,\n'
' "stddev" : '
'1992.233414,\n'
' "N" : 440456,\n'
' "percentile" : {\n'
' "1.000000" : 12096,\n'
' "5.000000" : 12224,\n'
' "10.000000" : 12224,\n'
' "20.000000" : 12224,\n'
' "30.000000" : 12224,\n'
' "40.000000" : 12224,\n'
' "50.000000" : 12352,\n'
' "60.000000" : 12352,\n'
' "70.000000" : 12352,\n'
' "80.000000" : 12352,\n'
' "90.000000" : 12352,\n'
' "95.000000" : 12608,\n'
' "99.000000" : 22400,\n'
' "99.500000" : 27520,\n'
' "99.900000" : 36608,\n'
' "99.950000" : 49920,\n'
' "99.990000" : 67072\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11952,\n'
' "max" : 108593,\n'
' "mean" : 15214.736416,\n'
' "stddev" : '
'2006.232859,\n'
' "N" : 440456\n'
' },\n'
' "bw_min" : 250789,\n'
' "bw_max" : 252440,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'251748.785714,\n'
' "bw_dev" : 511.742450,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 62697,\n'
' "iops_max" : 63110,\n'
' "iops_mean" : '
'62937.142857,\n'
' "iops_stddev" : '
'127.995106,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 9.485714,\n'
' "sys_cpu" : 38.314286,\n'
' "ctx" : 440441,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 98.659571,\n'
' "50" : 1.284351,\n'
' "100" : 0.047905,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 623550,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 5990,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 5990,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=84848: Tue '
'Oct 25 23:25:57 2022\n'
' read: IOPS=62.9k, BW=246MiB/s '
'(258MB/s)(1721MiB/7000msec)\n'
' slat (nsec): min=2465, '
'max=34736, avg=2646.84, '
'stdev=213.39\n'
' clat (nsec): min=240, '
'max=105898, avg=12567.90, '
'stdev=1992.23\n'
' lat (usec): min=11, max=108, '
'avg=15.21, stdev= 2.01\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12096], '
'5.00th=[12224], 10.00th=[12224], '
'20.00th=[12224],\n'
' | 30.00th=[12224], '
'40.00th=[12224], 50.00th=[12352], '
'60.00th=[12352],\n'
' | 70.00th=[12352], '
'80.00th=[12352], 90.00th=[12352], '
'95.00th=[12608],\n'
' | 99.00th=[22400], '
'99.50th=[27520], 99.90th=[36608], '
'99.95th=[49920],\n'
' | 99.99th=[67072]\n'
' bw ( KiB/s): min=250789, '
'max=252440, per=100.00%, '
'avg=251748.79, stdev=511.74, '
'samples=14\n'
' iops : min=62697, '
'max=63110, avg=62937.14, '
'stdev=128.00, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=98.66%, 50=1.28%, '
'100=0.05%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=9.49%, '
'sys=38.31%, ctx=440441, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=440456,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=246MiB/s (258MB/s), '
'246MiB/s-246MiB/s '
'(258MB/s-258MB/s), io=1721MiB '
'(1804MB), run=7000-7000msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=623550/0, '
'merge=0/0, ticks=5990/0, '
'in_queue=5990, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_36.output'),
'state': {'begin': 1666726067.5289435,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7989840507507324,
'end': 1666726068.3279276,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_36.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_36.state')},
'cmd_360': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734706,\n'
' "timestamp_ms" : '
'1666734706937,\n'
' "time" : "Tue Oct 25 23:51:46 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6417715200,\n'
' "io_kbytes" : 6267300,\n'
' "bw_bytes" : 916685502,\n'
' "bw" : 895200,\n'
' "iops" : 223799.028710,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1566817,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1292,\n'
' "max" : 57117,\n'
' "mean" : 2005.485569,\n'
' "stddev" : '
'861.647587,\n'
' "N" : 1566818\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 261,\n'
' "max" : 206247,\n'
' "mean" : '
'33462.330400,\n'
' "stddev" : '
'8878.873955,\n'
' "N" : 1566824,\n'
' "percentile" : {\n'
' "1.000000" : 23936,\n'
' "5.000000" : 28032,\n'
' "10.000000" : '
'28032,\n'
' "20.000000" : '
'28544,\n'
' "30.000000" : '
'30336,\n'
' "40.000000" : '
'30848,\n'
' "50.000000" : '
'31872,\n'
' "60.000000" : '
'32128,\n'
' "70.000000" : '
'32384,\n'
' "80.000000" : '
'32640,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'130560,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 16311,\n'
' "max" : 207599,\n'
' "mean" : '
'35467.815561,\n'
' "stddev" : '
'8771.771322,\n'
' "N" : 1566824\n'
' },\n'
' "bw_min" : 893368,\n'
' "bw_max" : 897939,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895407.071429,\n'
' "bw_dev" : 1490.748416,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223342,\n'
' "iops_max" : 224484,\n'
' "iops_mean" : '
'223851.714286,\n'
' "iops_stddev" : '
'372.323149,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.357143,\n'
' "sys_cpu" : 63.742857,\n'
' "ctx" : 541721,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 0.272080,\n'
' "50" : 92.033020,\n'
' "100" : 7.417778,\n'
' "250" : 0.277314,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=89597: Tue '
'Oct 25 23:51:46 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6120MiB/7001msec)\n'
' slat (nsec): min=1292, '
'max=57117, avg=2005.49, '
'stdev=861.65\n'
' clat (nsec): min=261, '
'max=206247, avg=33462.33, '
'stdev=8878.87\n'
' lat (usec): min=16, '
'max=207, avg=35.47, stdev= 8.77\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 24], '
'5.00th=[ 29], 10.00th=[ 29], '
'20.00th=[ 29],\n'
' | 30.00th=[ 31], '
'40.00th=[ 31], 50.00th=[ '
'32], 60.00th=[ 33],\n'
' | 70.00th=[ 33], '
'80.00th=[ 33], 90.00th=[ '
'46], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 68], 99.90th=[ '
'131], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=893368, '
'max=897939, per=100.00%, '
'avg=895407.07, stdev=1490.75, '
'samples=14\n'
' iops : min=223342, '
'max=224484, avg=223851.71, '
'stdev=372.32, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=0.27%, 50=92.03%, '
'100=7.42%\n'
' lat (usec) : 250=0.28%\n'
' cpu : usr=15.36%, '
'sys=63.74%, ctx=541721, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1566817,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6120MiB '
'(6418MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_360.output'),
'state': {'begin': 1666727617.4662776,
'cmd': 'cat '
'/tmp/fio-output_4K_8_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7870941162109375,
'end': 1666727618.2533717,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_360.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_360.state')},
'cmd_361': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_libaio-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_361.output'),
'state': {'begin': 1666727619.2475295,
'cmd': 'rm '
'/tmp/fio-output_4K_8_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7842357158660889,
'end': 1666727620.0317652,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_361.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_361.state')},
'cmd_362': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_362.output'),
'state': {'begin': 1666727620.033275,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_libaio-reference_0.txt"',
'cwd': 'None',
'elapsed': 11.068465232849121,
'end': 1666727631.1017401,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_362.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_362.state')},
'cmd_363': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=libaio, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734720,\n'
' "timestamp_ms" : '
'1666734720577,\n'
' "time" : "Tue Oct 25 23:52:00 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6421184512,\n'
' "io_kbytes" : 6270688,\n'
' "bw_bytes" : 917181047,\n'
' "bw" : 895684,\n'
' "iops" : 223920.011427,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567664,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1372,\n'
' "max" : 23945,\n'
' "mean" : 2044.074586,\n'
' "stddev" : '
'778.052273,\n'
' "N" : 1567665\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 15299,\n'
' "max" : 193182,\n'
' "mean" : '
'33405.325188,\n'
' "stddev" : '
'8836.893542,\n'
' "N" : 1567671,\n'
' "percentile" : {\n'
' "1.000000" : 24448,\n'
' "5.000000" : 28288,\n'
' "10.000000" : '
'28288,\n'
' "20.000000" : '
'28800,\n'
' "30.000000" : '
'29312,\n'
' "40.000000" : '
'30592,\n'
' "50.000000" : '
'31616,\n'
' "60.000000" : '
'32128,\n'
' "70.000000" : '
'32384,\n'
' "80.000000" : '
'33536,\n'
' "90.000000" : '
'44800,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'128512,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 17783,\n'
' "max" : 194615,\n'
' "mean" : '
'35449.399103,\n'
' "stddev" : '
'8754.063693,\n'
' "N" : 1567671\n'
' },\n'
' "bw_min" : 894608,\n'
' "bw_max" : 897456,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'896045.714286,\n'
' "bw_dev" : 800.433289,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223652,\n'
' "iops_max" : 224364,\n'
' "iops_mean" : '
'224011.428571,\n'
' "iops_stddev" : '
'200.229758,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 21.657143,\n'
' "sys_cpu" : 57.442857,\n'
' "ctx" : 533446,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999936,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.044142,\n'
' "50" : 92.606005,\n'
' "100" : 7.090231,\n'
' "250" : 0.260068,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2235295,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 71869,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 71869,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=89636: Tue '
'Oct 25 23:52:00 2022\n'
' read: IOPS=224k, BW=875MiB/s '
'(917MB/s)(6124MiB/7001msec)\n'
' slat (nsec): min=1372, '
'max=23945, avg=2044.07, '
'stdev=778.05\n'
' clat (usec): min=15, '
'max=193, avg=33.41, stdev= 8.84\n'
' lat (usec): min=17, '
'max=194, avg=35.45, stdev= 8.75\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 29], 10.00th=[ 29], '
'20.00th=[ 29],\n'
' | 30.00th=[ 30], '
'40.00th=[ 31], 50.00th=[ '
'32], 60.00th=[ 33],\n'
' | 70.00th=[ 33], '
'80.00th=[ 34], 90.00th=[ '
'45], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 68], 99.90th=[ '
'129], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=894608, '
'max=897456, per=100.00%, '
'avg=896045.71, stdev=800.43, '
'samples=14\n'
' iops : min=223652, '
'max=224364, avg=224011.43, '
'stdev=200.23, samples=14\n'
' lat (usec) : 20=0.04%, '
'50=92.61%, 100=7.09%, 250=0.26%\n'
' cpu : usr=21.66%, '
'sys=57.44%, ctx=533446, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567664,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=875MiB/s (917MB/s), '
'875MiB/s-875MiB/s '
'(917MB/s-917MB/s), io=6124MiB '
'(6421MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2235295/0, '
'merge=0/0, ticks=71869/0, '
'in_queue=71869, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_363.output'),
'state': {'begin': 1666727631.1085894,
'cmd': 'cat '
'/tmp/fio-output_4K_8_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.780418872833252,
'end': 1666727631.8890083,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_363.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_363.state')},
'cmd_364': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_libaio-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_364.output'),
'state': {'begin': 1666727632.8713968,
'cmd': 'rm '
'/tmp/fio-output_4K_8_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7804844379425049,
'end': 1666727633.6518812,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_364.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_364.state')},
'cmd_365': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_365.output'),
'state': {'begin': 1666727633.653573,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_libaio-reference_1.txt"',
'cwd': 'None',
'elapsed': 11.066412925720215,
'end': 1666727644.719986,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_365.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_365.state')},
'cmd_366': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=libaio, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734734,\n'
' "timestamp_ms" : '
'1666734734194,\n'
' "time" : "Tue Oct 25 23:52:14 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6419886080,\n'
' "io_kbytes" : 6269420,\n'
' "bw_bytes" : 916995583,\n'
' "bw" : 895503,\n'
' "iops" : 223874.875018,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567348,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1422,\n'
' "max" : 42911,\n'
' "mean" : 2070.390944,\n'
' "stddev" : '
'758.523721,\n'
' "N" : 1567348\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 391,\n'
' "max" : 177383,\n'
' "mean" : '
'33384.698160,\n'
' "stddev" : '
'8925.881402,\n'
' "N" : 1567355,\n'
' "percentile" : {\n'
' "1.000000" : 22912,\n'
' "5.000000" : 27264,\n'
' "10.000000" : '
'28544,\n'
' "20.000000" : '
'28800,\n'
' "30.000000" : '
'29056,\n'
' "40.000000" : '
'29568,\n'
' "50.000000" : '
'30848,\n'
' "60.000000" : '
'32128,\n'
' "70.000000" : '
'32640,\n'
' "80.000000" : '
'34048,\n'
' "90.000000" : '
'44800,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'128512,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 14407,\n'
' "max" : 178875,\n'
' "mean" : '
'35455.088327,\n'
' "stddev" : '
'8862.894253,\n'
' "N" : 1567355\n'
' },\n'
' "bw_min" : 893552,\n'
' "bw_max" : 897856,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895919.923077,\n'
' "bw_dev" : 1175.135344,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 223388,\n'
' "iops_max" : 224464,\n'
' "iops_mean" : '
'223979.923077,\n'
' "iops_stddev" : '
'293.956477,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 25.357143,\n'
' "sys_cpu" : 52.514286,\n'
' "ctx" : 563635,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999936,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.000000,\n'
' "20" : 0.058570,\n'
' "50" : 92.628185,\n'
' "100" : 7.051848,\n'
' "250" : 0.260950,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2177732,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 69367,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 69367,\n'
' "util" : 99.085923\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=89676: Tue '
'Oct 25 23:52:14 2022\n'
' read: IOPS=224k, BW=875MiB/s '
'(917MB/s)(6122MiB/7001msec)\n'
' slat (nsec): min=1422, '
'max=42911, avg=2070.39, '
'stdev=758.52\n'
' clat (nsec): min=391, '
'max=177383, avg=33384.70, '
'stdev=8925.88\n'
' lat (usec): min=14, '
'max=178, avg=35.46, stdev= 8.86\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 23], '
'5.00th=[ 28], 10.00th=[ 29], '
'20.00th=[ 29],\n'
' | 30.00th=[ 30], '
'40.00th=[ 30], 50.00th=[ '
'31], 60.00th=[ 33],\n'
' | 70.00th=[ 33], '
'80.00th=[ 35], 90.00th=[ '
'45], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 68], 99.90th=[ '
'129], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=893552, '
'max=897856, per=100.00%, '
'avg=895919.92, stdev=1175.14, '
'samples=13\n'
' iops : min=223388, '
'max=224464, avg=223979.92, '
'stdev=293.96, samples=13\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'20=0.06%, 50=92.63%, 100=7.05%, '
'250=0.26%\n'
' cpu : usr=25.36%, '
'sys=52.51%, ctx=563635, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567348,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=875MiB/s (917MB/s), '
'875MiB/s-875MiB/s '
'(917MB/s-917MB/s), io=6122MiB '
'(6420MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2177732/0, '
'merge=0/0, ticks=69367/0, '
'in_queue=69367, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_366.output'),
'state': {'begin': 1666727644.7220576,
'cmd': 'cat '
'/tmp/fio-output_4K_8_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.79469895362854,
'end': 1666727645.5167565,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_366.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_366.state')},
'cmd_367': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_libaio-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_367.output'),
'state': {'begin': 1666727646.5035763,
'cmd': 'rm '
'/tmp/fio-output_4K_8_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7806024551391602,
'end': 1666727647.2841787,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_367.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_367.state')},
'cmd_368': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_368.output'),
'state': {'begin': 1666727647.285832,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_libaio-reference_2.txt"',
'cwd': 'None',
'elapsed': 11.056997060775757,
'end': 1666727658.342829,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_368.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_368.state')},
'cmd_369': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=libaio, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734747,\n'
' "timestamp_ms" : '
'1666734747818,\n'
' "time" : "Tue Oct 25 23:52:27 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6419091456,\n'
' "io_kbytes" : 6268644,\n'
' "bw_bytes" : 916882081,\n'
' "bw" : 895392,\n'
' "iops" : 223847.164691,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567154,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1332,\n'
' "max" : 1297263,\n'
' "mean" : 2019.222772,\n'
' "stddev" : '
'1292.972535,\n'
' "N" : 1567154\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 12954,\n'
' "max" : 1329383,\n'
' "mean" : '
'33446.079906,\n'
' "stddev" : '
'9235.556673,\n'
' "N" : 1567161,\n'
' "percentile" : {\n'
' "1.000000" : 24192,\n'
' "5.000000" : 28032,\n'
' "10.000000" : '
'28288,\n'
' "20.000000" : '
'28800,\n'
' "30.000000" : '
'29568,\n'
' "40.000000" : '
'30592,\n'
' "50.000000" : '
'31616,\n'
' "60.000000" : '
'32128,\n'
' "70.000000" : '
'32384,\n'
' "80.000000" : '
'33536,\n'
' "90.000000" : '
'44800,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'129536,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 17462,\n'
' "max" : 1332139,\n'
' "mean" : '
'35465.301490,\n'
' "stddev" : '
'9217.281644,\n'
' "N" : 1567161\n'
' },\n'
' "bw_min" : 892584,\n'
' "bw_max" : 897699,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895743.142857,\n'
' "bw_dev" : 1372.428272,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223146,\n'
' "iops_max" : 224424,\n'
' "iops_mean" : '
'223935.714286,\n'
' "iops_stddev" : '
'343.066584,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 22.885714,\n'
' "sys_cpu" : 55.914286,\n'
' "ctx" : 540455,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999936,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.049261,\n'
' "50" : 92.557081,\n'
' "100" : 7.134079,\n'
' "250" : 0.259579,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2234710,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 71868,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 71868,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=89716: Tue '
'Oct 25 23:52:27 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6122MiB/7001msec)\n'
' slat (nsec): min=1332, '
'max=1297.3k, avg=2019.22, '
'stdev=1292.97\n'
' clat (usec): min=12, '
'max=1329, avg=33.45, stdev= '
'9.24\n'
' lat (usec): min=17, '
'max=1332, avg=35.47, stdev= '
'9.22\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 29], 10.00th=[ 29], '
'20.00th=[ 29],\n'
' | 30.00th=[ 30], '
'40.00th=[ 31], 50.00th=[ '
'32], 60.00th=[ 33],\n'
' | 70.00th=[ 33], '
'80.00th=[ 34], 90.00th=[ '
'45], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 68], 99.90th=[ '
'130], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=892584, '
'max=897699, per=100.00%, '
'avg=895743.14, stdev=1372.43, '
'samples=14\n'
' iops : min=223146, '
'max=224424, avg=223935.71, '
'stdev=343.07, samples=14\n'
' lat (usec) : 20=0.05%, '
'50=92.56%, 100=7.13%, 250=0.26%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=22.89%, '
'sys=55.91%, ctx=540455, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567154,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6122MiB '
'(6419MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2234710/0, '
'merge=0/0, ticks=71868/0, '
'in_queue=71868, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_369.output'),
'state': {'begin': 1666727658.3446329,
'cmd': 'cat '
'/tmp/fio-output_4K_8_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7802093029022217,
'end': 1666727659.1248422,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_369.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_369.state')},
'cmd_37': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_37.output'),
'state': {'begin': 1666726070.3615396,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.7853639125823975,
'end': 1666726071.1469035,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_37.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_37.state')},
'cmd_370': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_libaio-bdev_aio_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_370.output'),
'state': {'begin': 1666727661.132002,
'cmd': 'rm '
'/tmp/fio-output_4K_8_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.7778584957122803,
'end': 1666727661.9098606,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_370.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_370.state')},
'cmd_371': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_371.output'),
'state': {'begin': 1666727661.9113603,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_libaio-bdev_aio_0.txt"',
'cwd': 'None',
'elapsed': 11.515507459640503,
'end': 1666727673.4268677,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_371.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_371.state')},
'cmd_372': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734762,\n'
' "timestamp_ms" : '
'1666734762792,\n'
' "time" : "Tue Oct 25 23:52:42 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6415536128,\n'
' "io_kbytes" : 6265172,\n'
' "bw_bytes" : 916374250,\n'
' "bw" : 894896,\n'
' "iops" : 223723.039566,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1566285,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1553,\n'
' "max" : 184837,\n'
' "mean" : 1667.843817,\n'
' "stddev" : '
'320.534687,\n'
' "N" : 1566286\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 541,\n'
' "max" : 215374,\n'
' "mean" : '
'33789.432111,\n'
' "stddev" : '
'8671.015722,\n'
' "N" : 1566285,\n'
' "percentile" : {\n'
' "1.000000" : 24448,\n'
' "5.000000" : 30848,\n'
' "10.000000" : '
'30848,\n'
' "20.000000" : '
'31104,\n'
' "30.000000" : '
'31104,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31360,\n'
' "60.000000" : '
'31360,\n'
' "70.000000" : '
'31360,\n'
' "80.000000" : '
'31360,\n'
' "90.000000" : '
'43776,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62208,\n'
' "99.500000" : '
'66048,\n'
' "99.900000" : '
'134144,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 14217,\n'
' "max" : 217007,\n'
' "mean" : '
'35457.275931,\n'
' "stddev" : '
'8658.022173,\n'
' "N" : 1566285\n'
' },\n'
' "bw_min" : 893960,\n'
' "bw_max" : 896200,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895105.071429,\n'
' "bw_dev" : 806.860627,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223490,\n'
' "iops_max" : 224050,\n'
' "iops_mean" : '
'223776.214286,\n'
' "iops_stddev" : '
'201.682225,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 67.585714,\n'
' "sys_cpu" : 32.385714,\n'
' "ctx" : 14,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.100000,\n'
' "4" : 1.306531,\n'
' "8" : 98.689702,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.996755,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 0.047437,\n'
' "50" : 90.986251,\n'
' "100" : 8.674794,\n'
' "250" : 0.291326,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=89765: Tue '
'Oct 25 23:52:42 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(916MB/s)(6118MiB/7001msec)\n'
' slat (nsec): min=1553, '
'max=184837, avg=1667.84, '
'stdev=320.53\n'
' clat (nsec): min=541, '
'max=215374, avg=33789.43, '
'stdev=8671.02\n'
' lat (usec): min=14, '
'max=217, avg=35.46, stdev= 8.66\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 31], 10.00th=[ 31], '
'20.00th=[ 32],\n'
' | 30.00th=[ 32], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 32], '
'80.00th=[ 32], 90.00th=[ '
'44], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 67], 99.90th=[ '
'135], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=893960, '
'max=896200, per=100.00%, '
'avg=895105.07, stdev=806.86, '
'samples=14\n'
' iops : min=223490, '
'max=224050, avg=223776.21, '
'stdev=201.68, samples=14\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=0.05%, 50=90.99%, '
'100=8.67%\n'
' lat (usec) : 250=0.29%\n'
' cpu : usr=67.59%, '
'sys=32.39%, ctx=14, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=1.3%, 8=98.7%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1566285,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (916MB/s), '
'874MiB/s-874MiB/s '
'(916MB/s-916MB/s), io=6118MiB '
'(6416MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_372.output'),
'state': {'begin': 1666727673.4297333,
'cmd': 'cat '
'/tmp/fio-output_4K_8_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.7888956069946289,
'end': 1666727674.218629,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_372.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_372.state')},
'cmd_373': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_libaio-bdev_aio_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_373.output'),
'state': {'begin': 1666727676.2286882,
'cmd': 'rm '
'/tmp/fio-output_4K_8_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7789840698242188,
'end': 1666727677.0076723,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_373.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_373.state')},
'cmd_374': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_374.output'),
'state': {'begin': 1666727677.0096018,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_libaio-bdev_aio_1.txt"',
'cwd': 'None',
'elapsed': 11.522533655166626,
'end': 1666727688.5321355,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_374.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_374.state')},
'cmd_375': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734777,\n'
' "timestamp_ms" : '
'1666734777907,\n'
' "time" : "Tue Oct 25 23:52:57 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6419636224,\n'
' "io_kbytes" : 6269176,\n'
' "bw_bytes" : 916959894,\n'
' "bw" : 895468,\n'
' "iops" : 223866.161977,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567287,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1562,\n'
' "max" : 23695,\n'
' "mean" : 1655.664724,\n'
' "stddev" : '
'232.107596,\n'
' "N" : 1567287\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 7424,\n'
' "max" : 174908,\n'
' "mean" : '
'33773.507443,\n'
' "stddev" : '
'8548.878283,\n'
' "N" : 1567294,\n'
' "percentile" : {\n'
' "1.000000" : 24704,\n'
' "5.000000" : 30848,\n'
' "10.000000" : '
'31104,\n'
' "20.000000" : '
'31104,\n'
' "30.000000" : '
'31104,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31360,\n'
' "60.000000" : '
'31360,\n'
' "70.000000" : '
'31360,\n'
' "80.000000" : '
'31360,\n'
' "90.000000" : '
'43776,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62208,\n'
' "99.500000" : '
'66048,\n'
' "99.900000" : '
'130560,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 14487,\n'
' "max" : 176551,\n'
' "mean" : '
'35429.171972,\n'
' "stddev" : '
'8534.978594,\n'
' "N" : 1567294\n'
' },\n'
' "bw_min" : 892608,\n'
' "bw_max" : 898528,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895684.785714,\n'
' "bw_dev" : 1449.683476,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223152,\n'
' "iops_max" : 224632,\n'
' "iops_mean" : '
'223921.142857,\n'
' "iops_stddev" : '
'362.336617,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 68.328571,\n'
' "sys_cpu" : 31.642857,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.100000,\n'
' "4" : 1.275835,\n'
' "8" : 98.722952,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.997849,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 0.037517,\n'
' "50" : 91.020981,\n'
' "100" : 8.667525,\n'
' "250" : 0.274359,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=89814: Tue '
'Oct 25 23:52:57 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6122MiB/7001msec)\n'
' slat (nsec): min=1562, '
'max=23695, avg=1655.66, '
'stdev=232.11\n'
' clat (usec): min=7, max=174, '
'avg=33.77, stdev= 8.55\n'
' lat (usec): min=14, '
'max=176, avg=35.43, stdev= 8.53\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 31], 10.00th=[ 32], '
'20.00th=[ 32],\n'
' | 30.00th=[ 32], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 32], '
'80.00th=[ 32], 90.00th=[ '
'44], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 67], 99.90th=[ '
'131], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=892608, '
'max=898528, per=100.00%, '
'avg=895684.79, stdev=1449.68, '
'samples=14\n'
' iops : min=223152, '
'max=224632, avg=223921.14, '
'stdev=362.34, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=0.04%, 50=91.02%, 100=8.67%, '
'250=0.27%\n'
' cpu : usr=68.33%, '
'sys=31.64%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=1.3%, 8=98.7%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567287,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6122MiB '
'(6420MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_375.output'),
'state': {'begin': 1666727688.5368567,
'cmd': 'cat '
'/tmp/fio-output_4K_8_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7866065502166748,
'end': 1666727689.3234632,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_375.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_375.state')},
'cmd_376': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_libaio-bdev_aio_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_376.output'),
'state': {'begin': 1666727691.3690376,
'cmd': 'rm '
'/tmp/fio-output_4K_8_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.7804043292999268,
'end': 1666727692.149442,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_376.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_376.state')},
'cmd_377': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_377.output'),
'state': {'begin': 1666727692.1515708,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_libaio-bdev_aio_2.txt"',
'cwd': 'None',
'elapsed': 11.519450426101685,
'end': 1666727703.6710212,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_377.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_377.state')},
'cmd_378': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734793,\n'
' "timestamp_ms" : '
'1666734793038,\n'
' "time" : "Tue Oct 25 23:53:13 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6418944000,\n'
' "io_kbytes" : 6268500,\n'
' "bw_bytes" : 916861019,\n'
' "bw" : 895372,\n'
' "iops" : 223841.879731,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567117,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1542,\n'
' "max" : 51917,\n'
' "mean" : 1638.791215,\n'
' "stddev" : '
'252.351930,\n'
' "N" : 1567118\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 12954,\n'
' "max" : 173766,\n'
' "mean" : '
'33817.441936,\n'
' "stddev" : '
'8542.652012,\n'
' "N" : 1567117,\n'
' "percentile" : {\n'
' "1.000000" : 24448,\n'
' "5.000000" : 30848,\n'
' "10.000000" : '
'31104,\n'
' "20.000000" : '
'31104,\n'
' "30.000000" : '
'31104,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31360,\n'
' "60.000000" : '
'31360,\n'
' "70.000000" : '
'31360,\n'
' "80.000000" : '
'31616,\n'
' "90.000000" : '
'43776,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62208,\n'
' "99.500000" : '
'66048,\n'
' "99.900000" : '
'129536,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 15129,\n'
' "max" : 175399,\n'
' "mean" : '
'35456.233161,\n'
' "stddev" : '
'8527.216217,\n'
' "N" : 1567117\n'
' },\n'
' "bw_min" : 894312,\n'
' "bw_max" : 896672,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895574.142857,\n'
' "bw_dev" : 761.594467,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223578,\n'
' "iops_max" : 224168,\n'
' "iops_mean" : '
'223893.500000,\n'
' "iops_stddev" : '
'190.409579,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 71.014286,\n'
' "sys_cpu" : 28.971429,\n'
' "ctx" : 46,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.100000,\n'
' "4" : 1.324087,\n'
' "8" : 98.671446,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.995953,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.057941,\n'
' "50" : 90.970872,\n'
' "100" : 8.694820,\n'
' "250" : 0.276367,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=89863: Tue '
'Oct 25 23:53:13 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6122MiB/7001msec)\n'
' slat (nsec): min=1542, '
'max=51917, avg=1638.79, '
'stdev=252.35\n'
' clat (usec): min=12, '
'max=173, avg=33.82, stdev= 8.54\n'
' lat (usec): min=15, '
'max=175, avg=35.46, stdev= 8.53\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 31], 10.00th=[ 32], '
'20.00th=[ 32],\n'
' | 30.00th=[ 32], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 32], '
'80.00th=[ 32], 90.00th=[ '
'44], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 67], 99.90th=[ '
'130], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=894312, '
'max=896672, per=100.00%, '
'avg=895574.14, stdev=761.59, '
'samples=14\n'
' iops : min=223578, '
'max=224168, avg=223893.50, '
'stdev=190.41, samples=14\n'
' lat (usec) : 20=0.06%, '
'50=90.97%, 100=8.69%, 250=0.28%\n'
' cpu : usr=71.01%, '
'sys=28.97%, ctx=46, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=1.3%, 8=98.7%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567117,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6122MiB '
'(6419MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_378.output'),
'state': {'begin': 1666727703.6730046,
'cmd': 'cat '
'/tmp/fio-output_4K_8_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.7829241752624512,
'end': 1666727704.4559288,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_378.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_378.state')},
'cmd_379': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_379.output'),
'state': {'begin': 1666727706.523488,
'cmd': 'rm '
'/tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7925660610198975,
'end': 1666727707.316054,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_379.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_379.state')},
'cmd_38': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_38.output'),
'state': {'begin': 1666726071.149084,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt"',
'cwd': 'None',
'elapsed': 11.456365823745728,
'end': 1666726082.60545,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_38.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_38.state')},
'cmd_380': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_380.output'),
'state': {'begin': 1666727707.3180034,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.541755437850952,
'end': 1666727718.8597589,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_380.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_380.state')},
'cmd_381': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734808,\n'
' "timestamp_ms" : '
'1666734808232,\n'
' "time" : "Tue Oct 25 23:53:28 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6424309760,\n'
' "io_kbytes" : 6273740,\n'
' "bw_bytes" : 917627447,\n'
' "bw" : 896120,\n'
' "iops" : 224028.995858,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1568427,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1683,\n'
' "max" : 37610,\n'
' "mean" : 1920.989116,\n'
' "stddev" : '
'147.325480,\n'
' "N" : 1568428\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1113,\n'
' "max" : 185858,\n'
' "mean" : '
'33505.838697,\n'
' "stddev" : '
'8743.561843,\n'
' "N" : 1568430,\n'
' "percentile" : {\n'
' "1.000000" : 23168,\n'
' "5.000000" : 28288,\n'
' "10.000000" : '
'30336,\n'
' "20.000000" : '
'30592,\n'
' "30.000000" : '
'30848,\n'
' "40.000000" : '
'30848,\n'
' "50.000000" : '
'30848,\n'
' "60.000000" : '
'31104,\n'
' "70.000000" : '
'31104,\n'
' "80.000000" : '
'31360,\n'
' "90.000000" : '
'44800,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62208,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'130560,\n'
' "99.950000" : '
'142336,\n'
' "99.990000" : '
'160768\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 13776,\n'
' "max" : 187792,\n'
' "mean" : '
'35426.827932,\n'
' "stddev" : '
'8743.318794,\n'
' "N" : 1568430\n'
' },\n'
' "bw_min" : 894136,\n'
' "bw_max" : 898096,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'896613.785714,\n'
' "bw_dev" : 1166.083001,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223534,\n'
' "iops_max" : 224524,\n'
' "iops_mean" : '
'224153.357143,\n'
' "iops_stddev" : '
'291.452492,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 41.528571,\n'
' "sys_cpu" : 58.442857,\n'
' "ctx" : 18,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.100000,\n'
' "4" : 0.739403,\n'
' "8" : 99.259832,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999232,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.127963,\n'
' "50" : 91.673441,\n'
' "100" : 7.954977,\n'
' "250" : 0.243747,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=89912: Tue '
'Oct 25 23:53:28 2022\n'
' read: IOPS=224k, BW=875MiB/s '
'(918MB/s)(6127MiB/7001msec)\n'
' slat (nsec): min=1683, '
'max=37610, avg=1920.99, '
'stdev=147.33\n'
' clat (nsec): min=1113, '
'max=185858, avg=33505.84, '
'stdev=8743.56\n'
' lat (usec): min=13, '
'max=187, avg=35.43, stdev= 8.74\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 24], '
'5.00th=[ 29], 10.00th=[ 31], '
'20.00th=[ 31],\n'
' | 30.00th=[ 31], '
'40.00th=[ 31], 50.00th=[ '
'31], 60.00th=[ 32],\n'
' | 70.00th=[ 32], '
'80.00th=[ 32], 90.00th=[ '
'45], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 68], 99.90th=[ '
'131], 99.95th=[ 143],\n'
' | 99.99th=[ 161]\n'
' bw ( KiB/s): min=894136, '
'max=898096, per=100.00%, '
'avg=896613.79, stdev=1166.08, '
'samples=14\n'
' iops : min=223534, '
'max=224524, avg=224153.36, '
'stdev=291.45, samples=14\n'
' lat (usec) : 2=0.01%, '
'20=0.13%, 50=91.67%, 100=7.95%, '
'250=0.24%\n'
' cpu : usr=41.53%, '
'sys=58.44%, ctx=18, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=0.7%, 8=99.3%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1568427,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=875MiB/s (918MB/s), '
'875MiB/s-875MiB/s '
'(918MB/s-918MB/s), io=6127MiB '
'(6424MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_381.output'),
'state': {'begin': 1666727718.8625243,
'cmd': 'cat '
'/tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7950255870819092,
'end': 1666727719.6575499,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_381.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_381.state')},
'cmd_382': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_382.output'),
'state': {'begin': 1666727721.717935,
'cmd': 'rm '
'/tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7921743392944336,
'end': 1666727722.5101094,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_382.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_382.state')},
'cmd_383': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_383.output'),
'state': {'begin': 1666727722.5123904,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.539591789245605,
'end': 1666727734.0519822,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_383.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_383.state')},
'cmd_384': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734823,\n'
' "timestamp_ms" : '
'1666734823417,\n'
' "time" : "Tue Oct 25 23:53:43 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6421917696,\n'
' "io_kbytes" : 6271404,\n'
' "bw_bytes" : 917285772,\n'
' "bw" : 895786,\n'
' "iops" : 223945.722040,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567844,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1623,\n'
' "max" : 74419,\n'
' "mean" : 1755.833047,\n'
' "stddev" : '
'331.760129,\n'
' "N" : 1567844\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1042,\n'
' "max" : 177283,\n'
' "mean" : '
'33637.009474,\n'
' "stddev" : '
'8459.789321,\n'
' "N" : 1567851,\n'
' "percentile" : {\n'
' "1.000000" : 24704,\n'
' "5.000000" : 30592,\n'
' "10.000000" : '
'30848,\n'
' "20.000000" : '
'30848,\n'
' "30.000000" : '
'31104,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31104,\n'
' "60.000000" : '
'31360,\n'
' "70.000000" : '
'31360,\n'
' "80.000000" : '
'31616,\n'
' "90.000000" : '
'43776,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62208,\n'
' "99.500000" : '
'64768,\n'
' "99.900000" : '
'127488,\n'
' "99.950000" : '
'138240,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 13115,\n'
' "max" : 178976,\n'
' "mean" : '
'35392.842113,\n'
' "stddev" : '
'8437.240273,\n'
' "N" : 1567851\n'
' },\n'
' "bw_min" : 894312,\n'
' "bw_max" : 897768,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'896021.142857,\n'
' "bw_dev" : 1070.104442,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223578,\n'
' "iops_max" : 224442,\n'
' "iops_mean" : '
'224005.285714,\n'
' "iops_stddev" : '
'267.558312,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 48.500000,\n'
' "sys_cpu" : 51.471429,\n'
' "ctx" : 25,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.100000,\n'
' "4" : 1.606537,\n'
' "8" : 98.385299,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.995820,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 0.055618,\n'
' "50" : 91.209266,\n'
' "100" : 8.477757,\n'
' "250" : 0.257360,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=89963: Tue '
'Oct 25 23:53:43 2022\n'
' read: IOPS=224k, BW=875MiB/s '
'(917MB/s)(6124MiB/7001msec)\n'
' slat (nsec): min=1623, '
'max=74419, avg=1755.83, '
'stdev=331.76\n'
' clat (nsec): min=1042, '
'max=177283, avg=33637.01, '
'stdev=8459.79\n'
' lat (usec): min=13, '
'max=178, avg=35.39, stdev= 8.44\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 31], 10.00th=[ 31], '
'20.00th=[ 31],\n'
' | 30.00th=[ 32], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 32], '
'80.00th=[ 32], 90.00th=[ '
'44], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 65], 99.90th=[ '
'128], 99.95th=[ 139],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=894312, '
'max=897768, per=100.00%, '
'avg=896021.14, stdev=1070.10, '
'samples=14\n'
' iops : min=223578, '
'max=224442, avg=224005.29, '
'stdev=267.56, samples=14\n'
' lat (usec) : 2=0.01%, '
'10=0.01%, 20=0.06%, 50=91.21%, '
'100=8.48%\n'
' lat (usec) : 250=0.26%\n'
' cpu : usr=48.50%, '
'sys=51.47%, ctx=25, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=1.6%, 8=98.4%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567844,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=875MiB/s (917MB/s), '
'875MiB/s-875MiB/s '
'(917MB/s-917MB/s), io=6124MiB '
'(6422MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_384.output'),
'state': {'begin': 1666727734.0538235,
'cmd': 'cat '
'/tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7862935066223145,
'end': 1666727734.840117,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_384.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_384.state')},
'cmd_385': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_385.output'),
'state': {'begin': 1666727736.853087,
'cmd': 'rm '
'/tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7882089614868164,
'end': 1666727737.641296,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_385.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_385.state')},
'cmd_386': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_386.output'),
'state': {'begin': 1666727737.6432695,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.533258438110352,
'end': 1666727749.176528,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_386.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_386.state')},
'cmd_387': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=spdk_bdev, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734838,\n'
' "timestamp_ms" : '
'1666734838539,\n'
' "time" : "Tue Oct 25 23:53:58 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6419918848,\n'
' "io_kbytes" : 6269452,\n'
' "bw_bytes" : 917000263,\n'
' "bw" : 895508,\n'
' "iops" : 223876.017712,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1567356,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1583,\n'
' "max" : 180989,\n'
' "mean" : 1700.893440,\n'
' "stddev" : '
'355.436322,\n'
' "N" : 1567356\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1022,\n'
' "max" : 221516,\n'
' "mean" : '
'33718.714563,\n'
' "stddev" : '
'8534.695940,\n'
' "N" : 1567363,\n'
' "percentile" : {\n'
' "1.000000" : 24448,\n'
' "5.000000" : 30592,\n'
' "10.000000" : '
'30848,\n'
' "20.000000" : '
'31104,\n'
' "30.000000" : '
'31104,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31104,\n'
' "60.000000" : '
'31360,\n'
' "70.000000" : '
'31360,\n'
' "80.000000" : '
'31616,\n'
' "90.000000" : '
'43776,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62208,\n'
' "99.500000" : '
'65280,\n'
' "99.900000" : '
'129536,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 15730,\n'
' "max" : 223139,\n'
' "mean" : '
'35419.607731,\n'
' "stddev" : '
'8515.962916,\n'
' "N" : 1567363\n'
' },\n'
' "bw_min" : 894160,\n'
' "bw_max" : 898496,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895716.928571,\n'
' "bw_dev" : 996.908025,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223540,\n'
' "iops_max" : 224624,\n'
' "iops_mean" : '
'223929.357143,\n'
' "iops_stddev" : '
'249.118324,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 48.785714,\n'
' "sys_cpu" : 51.185714,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.100000,\n'
' "4" : 1.516120,\n'
' "8" : 98.479159,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.995950,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 0.054040,\n'
' "50" : 91.150256,\n'
' "100" : 8.527929,\n'
' "250" : 0.267967,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=90012: Tue '
'Oct 25 23:53:58 2022\n'
' read: IOPS=224k, BW=875MiB/s '
'(917MB/s)(6123MiB/7001msec)\n'
' slat (nsec): min=1583, '
'max=180989, avg=1700.89, '
'stdev=355.44\n'
' clat (nsec): min=1022, '
'max=221516, avg=33718.71, '
'stdev=8534.70\n'
' lat (usec): min=15, '
'max=223, avg=35.42, stdev= 8.52\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 25], '
'5.00th=[ 31], 10.00th=[ 31], '
'20.00th=[ 32],\n'
' | 30.00th=[ 32], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 32], '
'80.00th=[ 32], 90.00th=[ '
'44], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 66], 99.90th=[ '
'130], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=894160, '
'max=898496, per=100.00%, '
'avg=895716.93, stdev=996.91, '
'samples=14\n'
' iops : min=223540, '
'max=224624, avg=223929.36, '
'stdev=249.12, samples=14\n'
' lat (usec) : 2=0.01%, '
'10=0.01%, 20=0.05%, 50=91.15%, '
'100=8.53%\n'
' lat (usec) : 250=0.27%\n'
' cpu : usr=48.79%, '
'sys=51.19%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=1.5%, 8=98.5%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1567356,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=875MiB/s (917MB/s), '
'875MiB/s-875MiB/s '
'(917MB/s-917MB/s), io=6123MiB '
'(6420MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_387.output'),
'state': {'begin': 1666727749.1792693,
'cmd': 'cat '
'/tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7925474643707275,
'end': 1666727749.9718168,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_387.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_387.state')},
'cmd_388': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_libaio-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_388.output'),
'state': {'begin': 1666727750.9704022,
'cmd': 'rm '
'/tmp/fio-output_4K_8_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7865891456604004,
'end': 1666727751.7569914,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_388.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_388.state')},
'cmd_389': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_389.output'),
'state': {'begin': 1666727751.7590091,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_libaio-xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.096212387084961,
'end': 1666727762.8552215,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_389.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_389.state')},
'cmd_39': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733172,\n'
' "timestamp_ms" : '
'1666733172034,\n'
' "time" : "Tue Oct 25 23:26:12 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2409836544,\n'
' "io_kbytes" : 2353356,\n'
' "bw_bytes" : 344213190,\n'
' "bw" : 336145,\n'
' "iops" : 84036.423368,\n'
' "runtime" : 7001,\n'
' "total_ios" : 588339,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 160,\n'
' "max" : 10810,\n'
' "mean" : 173.542789,\n'
' "stddev" : 22.695499,\n'
' "N" : 588339\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 10410,\n'
' "max" : 78336,\n'
' "mean" : 11484.418524,\n'
' "stddev" : '
'2190.358012,\n'
' "N" : 588339,\n'
' "percentile" : {\n'
' "1.000000" : 10944,\n'
' "5.000000" : 10944,\n'
' "10.000000" : 11072,\n'
' "20.000000" : 11072,\n'
' "30.000000" : 11072,\n'
' "40.000000" : 11072,\n'
' "50.000000" : 11072,\n'
' "60.000000" : 11200,\n'
' "70.000000" : 11200,\n'
' "80.000000" : 11328,\n'
' "90.000000" : 11328,\n'
' "95.000000" : 11968,\n'
' "99.000000" : 22144,\n'
' "99.500000" : 28288,\n'
' "99.900000" : 28544,\n'
' "99.950000" : 43776,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10580,\n'
' "max" : 78517,\n'
' "mean" : 11657.961313,\n'
' "stddev" : '
'2190.707867,\n'
' "N" : 588339\n'
' },\n'
' "bw_min" : 335000,\n'
' "bw_max" : 337450,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'336330.692308,\n'
' "bw_dev" : 777.678960,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 83750,\n'
' "iops_max" : 84362,\n'
' "iops_mean" : '
'84082.615385,\n'
' "iops_stddev" : '
'194.345714,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 83.014286,\n'
' "sys_cpu" : 16.971429,\n'
' "ctx" : 16,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 98.617124,\n'
' "50" : 1.344293,\n'
' "100" : 0.038583,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=84897: Tue '
'Oct 25 23:26:12 2022\n'
' read: IOPS=84.0k, BW=328MiB/s '
'(344MB/s)(2298MiB/7001msec)\n'
' slat (nsec): min=160, '
'max=10810, avg=173.54, '
'stdev=22.70\n'
' clat (nsec): min=10410, '
'max=78336, avg=11484.42, '
'stdev=2190.36\n'
' lat (nsec): min=10580, '
'max=78517, avg=11657.96, '
'stdev=2190.71\n'
' clat percentiles (nsec):\n'
' | 1.00th=[10944], '
'5.00th=[10944], 10.00th=[11072], '
'20.00th=[11072],\n'
' | 30.00th=[11072], '
'40.00th=[11072], 50.00th=[11072], '
'60.00th=[11200],\n'
' | 70.00th=[11200], '
'80.00th=[11328], 90.00th=[11328], '
'95.00th=[11968],\n'
' | 99.00th=[22144], '
'99.50th=[28288], 99.90th=[28544], '
'99.95th=[43776],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=335000, '
'max=337450, per=100.00%, '
'avg=336330.69, stdev=777.68, '
'samples=13\n'
' iops : min=83750, '
'max=84362, avg=84082.62, '
'stdev=194.35, samples=13\n'
' lat (usec) : 20=98.62%, '
'50=1.34%, 100=0.04%\n'
' cpu : usr=83.01%, '
'sys=16.97%, ctx=16, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=588339,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=328MiB/s (344MB/s), '
'328MiB/s-328MiB/s '
'(344MB/s-344MB/s), io=2298MiB '
'(2410MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_39.output'),
'state': {'begin': 1666726082.608158,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.7873132228851318,
'end': 1666726083.3954713,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_39.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_39.state')},
'cmd_390': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734852,\n'
' "timestamp_ms" : '
'1666734852329,\n'
' "time" : "Tue Oct 25 23:54:12 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6413660160,\n'
' "io_kbytes" : 6263340,\n'
' "bw_bytes" : 916106293,\n'
' "bw" : 894635,\n'
' "iops" : 223657.620340,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1565827,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1463,\n'
' "max" : 42490,\n'
' "mean" : 1879.747679,\n'
' "stddev" : '
'476.780147,\n'
' "N" : 1565828\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2766,\n'
' "max" : 250209,\n'
' "mean" : '
'33625.378656,\n'
' "stddev" : '
'9412.417229,\n'
' "N" : 1565834,\n'
' "percentile" : {\n'
' "1.000000" : 22912,\n'
' "5.000000" : 25984,\n'
' "10.000000" : '
'27264,\n'
' "20.000000" : '
'28544,\n'
' "30.000000" : '
'29312,\n'
' "40.000000" : '
'30336,\n'
' "50.000000" : '
'31104,\n'
' "60.000000" : '
'31872,\n'
' "70.000000" : '
'33024,\n'
' "80.000000" : '
'36608,\n'
' "90.000000" : '
'45312,\n'
' "95.000000" : '
'52480,\n'
' "99.000000" : '
'63232,\n'
' "99.500000" : '
'69120,\n'
' "99.900000" : '
'134144,\n'
' "99.950000" : '
'142336,\n'
' "99.990000" : '
'160768\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 15689,\n'
' "max" : 262933,\n'
' "mean" : '
'35505.126049,\n'
' "stddev" : '
'9396.777144,\n'
' "N" : 1565834\n'
' },\n'
' "bw_min" : 887864,\n'
' "bw_max" : 897378,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895042.928571,\n'
' "bw_dev" : 2300.390016,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 221966,\n'
' "iops_max" : 224344,\n'
' "iops_mean" : '
'223760.500000,\n'
' "iops_stddev" : '
'574.996020,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 22.085714,\n'
' "sys_cpu" : 54.371429,\n'
' "ctx" : 539751,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.000000,\n'
' "20" : 0.414669,\n'
' "50" : 92.437734,\n'
' "100" : 6.875664,\n'
' "250" : 0.272188,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=90051: Tue '
'Oct 25 23:54:12 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(916MB/s)(6117MiB/7001msec)\n'
' slat (nsec): min=1463, '
'max=42490, avg=1879.75, '
'stdev=476.78\n'
' clat (usec): min=2, max=250, '
'avg=33.63, stdev= 9.41\n'
' lat (usec): min=15, '
'max=262, avg=35.51, stdev= 9.40\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 23], '
'5.00th=[ 26], 10.00th=[ 28], '
'20.00th=[ 29],\n'
' | 30.00th=[ 30], '
'40.00th=[ 31], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 34], '
'80.00th=[ 37], 90.00th=[ '
'46], 95.00th=[ 53],\n'
' | 99.00th=[ 64], '
'99.50th=[ 70], 99.90th=[ '
'135], 99.95th=[ 143],\n'
' | 99.99th=[ 161]\n'
' bw ( KiB/s): min=887864, '
'max=897378, per=100.00%, '
'avg=895042.93, stdev=2300.39, '
'samples=14\n'
' iops : min=221966, '
'max=224344, avg=223760.50, '
'stdev=575.00, samples=14\n'
' lat (usec) : 4=0.01%, '
'20=0.41%, 50=92.44%, 100=6.88%, '
'250=0.27%\n'
' lat (usec) : 500=0.01%\n'
' cpu : usr=22.09%, '
'sys=54.37%, ctx=539751, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1565827,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (916MB/s), '
'874MiB/s-874MiB/s '
'(916MB/s-916MB/s), io=6117MiB '
'(6414MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_390.output'),
'state': {'begin': 1666727762.8579476,
'cmd': 'cat '
'/tmp/fio-output_4K_8_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7954163551330566,
'end': 1666727763.653364,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_390.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_390.state')},
'cmd_391': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_libaio-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_391.output'),
'state': {'begin': 1666727764.653474,
'cmd': 'rm '
'/tmp/fio-output_4K_8_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.784369945526123,
'end': 1666727765.437844,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_391.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_391.state')},
'cmd_392': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_392.output'),
'state': {'begin': 1666727765.4397173,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_libaio-xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.105721712112427,
'end': 1666727776.545439,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_392.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_392.state')},
'cmd_393': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734866,\n'
' "timestamp_ms" : '
'1666734866018,\n'
' "time" : "Tue Oct 25 23:54:26 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6422921216,\n'
' "io_kbytes" : 6272384,\n'
' "bw_bytes" : 917429112,\n'
' "bw" : 895926,\n'
' "iops" : 223980.717040,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1568089,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1482,\n'
' "max" : 25568,\n'
' "mean" : 1774.253431,\n'
' "stddev" : '
'161.376328,\n'
' "N" : 1568089\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 13826,\n'
' "max" : 201578,\n'
' "mean" : '
'33675.984157,\n'
' "stddev" : '
'9110.646423,\n'
' "N" : 1568096,\n'
' "percentile" : {\n'
' "1.000000" : 23936,\n'
' "5.000000" : 25984,\n'
' "10.000000" : '
'27264,\n'
' "20.000000" : '
'29056,\n'
' "30.000000" : '
'30080,\n'
' "40.000000" : '
'30848,\n'
' "50.000000" : '
'31360,\n'
' "60.000000" : '
'32128,\n'
' "70.000000" : '
'33024,\n'
' "80.000000" : '
'35072,\n'
' "90.000000" : '
'44800,\n'
' "95.000000" : '
'52480,\n'
' "99.000000" : '
'62720,\n'
' "99.500000" : '
'68096,\n'
' "99.900000" : '
'130560,\n'
' "99.950000" : '
'142336,\n'
' "99.990000" : '
'158720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 15499,\n'
' "max" : 207278,\n'
' "mean" : '
'35450.237329,\n'
' "stddev" : '
'9103.650064,\n'
' "N" : 1568096\n'
' },\n'
' "bw_min" : 894984,\n'
' "bw_max" : 897680,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'896217.000000,\n'
' "bw_dev" : 865.985877,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223746,\n'
' "iops_max" : 224420,\n'
' "iops_mean" : '
'224054.214286,\n'
' "iops_stddev" : '
'216.539276,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 19.214286,\n'
' "sys_cpu" : 54.914286,\n'
' "ctx" : 652730,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.291055,\n'
' "50" : 92.258539,\n'
' "100" : 7.203864,\n'
' "250" : 0.246989,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=90091: Tue '
'Oct 25 23:54:26 2022\n'
' read: IOPS=224k, BW=875MiB/s '
'(917MB/s)(6125MiB/7001msec)\n'
' slat (nsec): min=1482, '
'max=25568, avg=1774.25, '
'stdev=161.38\n'
' clat (usec): min=13, '
'max=201, avg=33.68, stdev= 9.11\n'
' lat (usec): min=15, '
'max=207, avg=35.45, stdev= 9.10\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 24], '
'5.00th=[ 26], 10.00th=[ 28], '
'20.00th=[ 30],\n'
' | 30.00th=[ 31], '
'40.00th=[ 31], 50.00th=[ '
'32], 60.00th=[ 33],\n'
' | 70.00th=[ 34], '
'80.00th=[ 36], 90.00th=[ '
'45], 95.00th=[ 53],\n'
' | 99.00th=[ 63], '
'99.50th=[ 69], 99.90th=[ '
'131], 99.95th=[ 143],\n'
' | 99.99th=[ 159]\n'
' bw ( KiB/s): min=894984, '
'max=897680, per=100.00%, '
'avg=896217.00, stdev=865.99, '
'samples=14\n'
' iops : min=223746, '
'max=224420, avg=224054.21, '
'stdev=216.54, samples=14\n'
' lat (usec) : 20=0.29%, '
'50=92.26%, 100=7.20%, 250=0.25%\n'
' cpu : usr=19.21%, '
'sys=54.91%, ctx=652730, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1568089,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=875MiB/s (917MB/s), '
'875MiB/s-875MiB/s '
'(917MB/s-917MB/s), io=6125MiB '
'(6423MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_393.output'),
'state': {'begin': 1666727776.5485766,
'cmd': 'cat '
'/tmp/fio-output_4K_8_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7902369499206543,
'end': 1666727777.3388135,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_393.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_393.state')},
'cmd_394': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_8_libaio-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_394.output'),
'state': {'begin': 1666727778.3286088,
'cmd': 'rm '
'/tmp/fio-output_4K_8_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7883713245391846,
'end': 1666727779.11698,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_394.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_394.state')},
'cmd_395': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_395.output'),
'state': {'begin': 1666727779.1189435,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="4K" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_8_libaio-xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.085206508636475,
'end': 1666727790.20415,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_395.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_395.state')},
'cmd_396': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) '
'4096B-4096B, ioengine=xnvme, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666734879,\n'
' "timestamp_ms" : '
'1666734879675,\n'
' "time" : "Tue Oct 25 23:54:39 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'6417117184,\n'
' "io_kbytes" : 6266716,\n'
' "bw_bytes" : 916600083,\n'
' "bw" : 895117,\n'
' "iops" : 223778.317383,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1566672,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1452,\n'
' "max" : 42480,\n'
' "mean" : 2043.000890,\n'
' "stddev" : '
'749.595984,\n'
' "N" : 1566672\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 440,\n'
' "max" : 228789,\n'
' "mean" : '
'33399.843687,\n'
' "stddev" : '
'8968.879387,\n'
' "N" : 1566679,\n'
' "percentile" : {\n'
' "1.000000" : 23168,\n'
' "5.000000" : 27008,\n'
' "10.000000" : '
'27520,\n'
' "20.000000" : '
'28544,\n'
' "30.000000" : '
'29824,\n'
' "40.000000" : '
'31104,\n'
' "50.000000" : '
'31360,\n'
' "60.000000" : '
'31616,\n'
' "70.000000" : '
'32384,\n'
' "80.000000" : '
'34048,\n'
' "90.000000" : '
'44800,\n'
' "95.000000" : '
'51968,\n'
' "99.000000" : '
'62208,\n'
' "99.500000" : '
'67072,\n'
' "99.900000" : '
'132096,\n'
' "99.950000" : '
'140288,\n'
' "99.990000" : '
'156672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 18515,\n'
' "max" : 230592,\n'
' "mean" : '
'35442.843629,\n'
' "stddev" : '
'8913.795051,\n'
' "N" : 1566679\n'
' },\n'
' "bw_min" : 893096,\n'
' "bw_max" : 897528,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'895350.214286,\n'
' "bw_dev" : 1168.143970,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 223274,\n'
' "iops_max" : 224382,\n'
' "iops_mean" : '
'223837.500000,\n'
' "iops_stddev" : '
'292.004413,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 23.857143,\n'
' "sys_cpu" : 59.200000,\n'
' "ctx" : 370799,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999968,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 0.034468,\n'
' "50" : 92.821854,\n'
' "100" : 6.875274,\n'
' "250" : 0.268595,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=90132: Tue '
'Oct 25 23:54:39 2022\n'
' read: IOPS=224k, BW=874MiB/s '
'(917MB/s)(6120MiB/7001msec)\n'
' slat (nsec): min=1452, '
'max=42480, avg=2043.00, '
'stdev=749.60\n'
' clat (nsec): min=440, '
'max=228789, avg=33399.84, '
'stdev=8968.88\n'
' lat (usec): min=18, '
'max=230, avg=35.44, stdev= 8.91\n'
' clat percentiles (usec):\n'
' | 1.00th=[ 24], '
'5.00th=[ 28], 10.00th=[ 28], '
'20.00th=[ 29],\n'
' | 30.00th=[ 30], '
'40.00th=[ 32], 50.00th=[ '
'32], 60.00th=[ 32],\n'
' | 70.00th=[ 33], '
'80.00th=[ 35], 90.00th=[ '
'45], 95.00th=[ 52],\n'
' | 99.00th=[ 63], '
'99.50th=[ 68], 99.90th=[ '
'133], 99.95th=[ 141],\n'
' | 99.99th=[ 157]\n'
' bw ( KiB/s): min=893096, '
'max=897528, per=100.00%, '
'avg=895350.21, stdev=1168.14, '
'samples=14\n'
' iops : min=223274, '
'max=224382, avg=223837.50, '
'stdev=292.00, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=0.03%, 50=92.82%, '
'100=6.88%\n'
' lat (usec) : 250=0.27%\n'
' cpu : usr=23.86%, '
'sys=59.20%, ctx=370799, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1566672,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=874MiB/s (917MB/s), '
'874MiB/s-874MiB/s '
'(917MB/s-917MB/s), io=6120MiB '
'(6417MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_396.output'),
'state': {'begin': 1666727790.2068746,
'cmd': 'cat '
'/tmp/fio-output_4K_8_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7910661697387695,
'end': 1666727790.9979408,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_396.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_396.state')},
'cmd_40': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_40.output'),
'state': {'begin': 1666726085.471208,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7902677059173584,
'end': 1666726086.2614758,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_40.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_40.state')},
'cmd_41': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_41.output'),
'state': {'begin': 1666726086.2634542,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt"',
'cwd': 'None',
'elapsed': 11.455034732818604,
'end': 1666726097.718489,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_41.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_41.state')},
'cmd_42': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733187,\n'
' "timestamp_ms" : '
'1666733187146,\n'
' "time" : "Tue Oct 25 23:26:27 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2400022528,\n'
' "io_kbytes" : 2343772,\n'
' "bw_bytes" : 342811388,\n'
' "bw" : 334776,\n'
' "iops" : 83694.186545,\n'
' "runtime" : 7001,\n'
' "total_ios" : 585943,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 160,\n'
' "max" : 9938,\n'
' "mean" : 176.388493,\n'
' "stddev" : 26.408079,\n'
' "N" : 585943\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 10430,\n'
' "max" : 113473,\n'
' "mean" : 11527.381930,\n'
' "stddev" : '
'2234.725223,\n'
' "N" : 585943,\n'
' "percentile" : {\n'
' "1.000000" : 10944,\n'
' "5.000000" : 11072,\n'
' "10.000000" : 11072,\n'
' "20.000000" : 11072,\n'
' "30.000000" : 11072,\n'
' "40.000000" : 11072,\n'
' "50.000000" : 11200,\n'
' "60.000000" : 11200,\n'
' "70.000000" : 11328,\n'
' "80.000000" : 11328,\n'
' "90.000000" : 11456,\n'
' "95.000000" : 12096,\n'
' "99.000000" : 22400,\n'
' "99.500000" : 28288,\n'
' "99.900000" : 28800,\n'
' "99.950000" : 44288,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10610,\n'
' "max" : 113653,\n'
' "mean" : 11703.770423,\n'
' "stddev" : '
'2235.118972,\n'
' "N" : 585943\n'
' },\n'
' "bw_min" : 334472,\n'
' "bw_max" : 336208,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'334961.142857,\n'
' "bw_dev" : 434.449228,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 83618,\n'
' "iops_max" : 84052,\n'
' "iops_mean" : '
'83740.214286,\n'
' "iops_stddev" : '
'108.614476,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 80.728571,\n'
' "sys_cpu" : 19.257143,\n'
' "ctx" : 15,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 98.614541,\n'
' "50" : 1.340915,\n'
' "100" : 0.044373,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=84947: Tue '
'Oct 25 23:26:27 2022\n'
' read: IOPS=83.7k, BW=327MiB/s '
'(343MB/s)(2289MiB/7001msec)\n'
' slat (nsec): min=160, '
'max=9938, avg=176.39, '
'stdev=26.41\n'
' clat (usec): min=10, max=113, '
'avg=11.53, stdev= 2.23\n'
' lat (usec): min=10, max=113, '
'avg=11.70, stdev= 2.24\n'
' clat percentiles (nsec):\n'
' | 1.00th=[10944], '
'5.00th=[11072], 10.00th=[11072], '
'20.00th=[11072],\n'
' | 30.00th=[11072], '
'40.00th=[11072], 50.00th=[11200], '
'60.00th=[11200],\n'
' | 70.00th=[11328], '
'80.00th=[11328], 90.00th=[11456], '
'95.00th=[12096],\n'
' | 99.00th=[22400], '
'99.50th=[28288], 99.90th=[28800], '
'99.95th=[44288],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=334472, '
'max=336208, per=100.00%, '
'avg=334961.14, stdev=434.45, '
'samples=14\n'
' iops : min=83618, '
'max=84052, avg=83740.21, '
'stdev=108.61, samples=14\n'
' lat (usec) : 20=98.61%, '
'50=1.34%, 100=0.04%, 250=0.01%\n'
' cpu : usr=80.73%, '
'sys=19.26%, ctx=15, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=585943,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=327MiB/s (343MB/s), '
'327MiB/s-327MiB/s '
'(343MB/s-343MB/s), io=2289MiB '
'(2400MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_42.output'),
'state': {'begin': 1666726097.7213132,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7839422225952148,
'end': 1666726098.5052555,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_42.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_42.state')},
'cmd_43': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_43.output'),
'state': {'begin': 1666726100.5752542,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.790607213973999,
'end': 1666726101.3658614,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_43.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_43.state')},
'cmd_44': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_44.output'),
'state': {'begin': 1666726101.367674,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt"',
'cwd': 'None',
'elapsed': 11.460201978683472,
'end': 1666726112.827876,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_44.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_44.state')},
'cmd_45': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733202,\n'
' "timestamp_ms" : '
'1666733202256,\n'
' "time" : "Tue Oct 25 23:26:42 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2856644608,\n'
' "io_kbytes" : 2789692,\n'
' "bw_bytes" : 408033796,\n'
' "bw" : 398470,\n'
' "iops" : 99617.626053,\n'
' "runtime" : 7001,\n'
' "total_ios" : 697423,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 161,\n'
' "max" : 12935,\n'
' "mean" : 181.008962,\n'
' "stddev" : 43.708701,\n'
' "N" : 697423\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 9037,\n'
' "max" : 95539,\n'
' "mean" : 9612.561520,\n'
' "stddev" : '
'1943.039396,\n'
' "N" : 697423,\n'
' "percentile" : {\n'
' "1.000000" : 9152,\n'
' "5.000000" : 9152,\n'
' "10.000000" : 9152,\n'
' "20.000000" : 9152,\n'
' "30.000000" : 9280,\n'
' "40.000000" : 9280,\n'
' "50.000000" : 9280,\n'
' "60.000000" : 9408,\n'
' "70.000000" : 9408,\n'
' "80.000000" : 9408,\n'
' "90.000000" : 9408,\n'
' "95.000000" : 10048,\n'
' "99.000000" : 18816,\n'
' "99.500000" : 20864,\n'
' "99.900000" : 25472,\n'
' "99.950000" : 42752,\n'
' "99.990000" : 67072\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9218,\n'
' "max" : 95719,\n'
' "mean" : 9793.570482,\n'
' "stddev" : '
'1943.679350,\n'
' "N" : 697423\n'
' },\n'
' "bw_min" : 397920,\n'
' "bw_max" : 399024,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'398570.642857,\n'
' "bw_dev" : 358.930807,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 99480,\n'
' "iops_max" : 99756,\n'
' "iops_mean" : '
'99642.642857,\n'
' "iops_stddev" : '
'89.712024,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 70.271429,\n'
' "sys_cpu" : 29.700000,\n'
' "ctx" : 8,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 94.961594,\n'
' "20" : 4.229141,\n'
' "50" : 0.766393,\n'
' "100" : 0.042872,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=84996: Tue '
'Oct 25 23:26:42 2022\n'
' read: IOPS=99.6k, BW=389MiB/s '
'(408MB/s)(2724MiB/7001msec)\n'
' slat (nsec): min=161, '
'max=12935, avg=181.01, '
'stdev=43.71\n'
' clat (nsec): min=9037, '
'max=95539, avg=9612.56, '
'stdev=1943.04\n'
' lat (nsec): min=9218, '
'max=95719, avg=9793.57, '
'stdev=1943.68\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9152], 5.00th=[ '
'9152], 10.00th=[ 9152], 20.00th=[ '
'9152],\n'
' | 30.00th=[ 9280], 40.00th=[ '
'9280], 50.00th=[ 9280], 60.00th=[ '
'9408],\n'
' | 70.00th=[ 9408], 80.00th=[ '
'9408], 90.00th=[ 9408], '
'95.00th=[10048],\n'
' | 99.00th=[18816], '
'99.50th=[20864], 99.90th=[25472], '
'99.95th=[42752],\n'
' | 99.99th=[67072]\n'
' bw ( KiB/s): min=397920, '
'max=399024, per=100.00%, '
'avg=398570.64, stdev=358.93, '
'samples=14\n'
' iops : min=99480, '
'max=99756, avg=99642.64, '
'stdev=89.71, samples=14\n'
' lat (usec) : 10=94.96%, '
'20=4.23%, 50=0.77%, 100=0.04%\n'
' cpu : usr=70.27%, '
'sys=29.70%, ctx=8, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=697423,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=389MiB/s (408MB/s), '
'389MiB/s-389MiB/s '
'(408MB/s-408MB/s), io=2724MiB '
'(2857MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_45.output'),
'state': {'begin': 1666726112.8305573,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.7889697551727295,
'end': 1666726113.619527,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_45.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_45.state')},
'cmd_46': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_46.output'),
'state': {'begin': 1666726115.6897912,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7863481044769287,
'end': 1666726116.4761393,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_46.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_46.state')},
'cmd_47': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_47.output'),
'state': {'begin': 1666726116.478392,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.34797716140747,
'end': 1666726127.826369,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_47.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_47.state')},
'cmd_48': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733217,\n'
' "timestamp_ms" : '
'1666733217255,\n'
' "time" : "Tue Oct 25 23:26:57 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2650079232,\n'
' "io_kbytes" : 2587968,\n'
' "bw_bytes" : 378528671,\n'
' "bw" : 369656,\n'
' "iops" : 92414.226539,\n'
' "runtime" : 7001,\n'
' "total_ios" : 646992,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1673,\n'
' "max" : 15529,\n'
' "mean" : 1800.539421,\n'
' "stddev" : 116.222659,\n'
' "N" : 646992\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 251,\n'
' "max" : 115536,\n'
' "mean" : 8778.187362,\n'
' "stddev" : '
'1981.005506,\n'
' "N" : 646992,\n'
' "percentile" : {\n'
' "1.000000" : 8256,\n'
' "5.000000" : 8384,\n'
' "10.000000" : 8384,\n'
' "20.000000" : 8384,\n'
' "30.000000" : 8384,\n'
' "40.000000" : 8384,\n'
' "50.000000" : 8512,\n'
' "60.000000" : 8512,\n'
' "70.000000" : 8512,\n'
' "80.000000" : 8512,\n'
' "90.000000" : 8640,\n'
' "95.000000" : 9408,\n'
' "99.000000" : 18816,\n'
' "99.500000" : 20352,\n'
' "99.900000" : 24960,\n'
' "99.950000" : 41728,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9598,\n'
' "max" : 117290,\n'
' "mean" : 10578.726783,\n'
' "stddev" : '
'1983.792726,\n'
' "N" : 646992\n'
' },\n'
' "bw_min" : 367720,\n'
' "bw_max" : 370808,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'369810.769231,\n'
' "bw_dev" : 877.410504,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 91930,\n'
' "iops_max" : 92702,\n'
' "iops_mean" : '
'92452.615385,\n'
' "iops_stddev" : '
'219.323406,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 5.642857,\n'
' "sys_cpu" : 94.328571,\n'
' "ctx" : 14,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 95.804121,\n'
' "20" : 3.551049,\n'
' "50" : 0.603562,\n'
' "100" : 0.036940,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=85046: Tue '
'Oct 25 23:26:57 2022\n'
' read: IOPS=92.4k, BW=361MiB/s '
'(379MB/s)(2527MiB/7001msec)\n'
' slat (nsec): min=1673, '
'max=15529, avg=1800.54, '
'stdev=116.22\n'
' clat (nsec): min=251, '
'max=115536, avg=8778.19, '
'stdev=1981.01\n'
' lat (usec): min=9, max=117, '
'avg=10.58, stdev= 1.98\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8256], 5.00th=[ '
'8384], 10.00th=[ 8384], 20.00th=[ '
'8384],\n'
' | 30.00th=[ 8384], 40.00th=[ '
'8384], 50.00th=[ 8512], 60.00th=[ '
'8512],\n'
' | 70.00th=[ 8512], 80.00th=[ '
'8512], 90.00th=[ 8640], 95.00th=[ '
'9408],\n'
' | 99.00th=[18816], '
'99.50th=[20352], 99.90th=[24960], '
'99.95th=[41728],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=367720, '
'max=370808, per=100.00%, '
'avg=369810.77, stdev=877.41, '
'samples=13\n'
' iops : min=91930, '
'max=92702, avg=92452.62, '
'stdev=219.32, samples=13\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=95.80%, 20=3.55%, 50=0.60%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=5.64%, '
'sys=94.33%, ctx=14, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=646992,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=361MiB/s (379MB/s), '
'361MiB/s-361MiB/s '
'(379MB/s-379MB/s), io=2527MiB '
'(2650MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_48.output'),
'state': {'begin': 1666726127.829114,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7902984619140625,
'end': 1666726128.6194124,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_48.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_48.state')},
'cmd_49': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_49.output'),
'state': {'begin': 1666726130.6240525,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7834982872009277,
'end': 1666726131.4075508,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_49.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_49.state')},
'cmd_50': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_50.output'),
'state': {'begin': 1666726131.4095988,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.450175523757935,
'end': 1666726142.8597744,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_50.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_50.state')},
'cmd_51': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733232,\n'
' "timestamp_ms" : '
'1666733232288,\n'
' "time" : "Tue Oct 25 23:27:12 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2934267904,\n'
' "io_kbytes" : 2865496,\n'
' "bw_bytes" : 419121254,\n'
' "bw" : 409298,\n'
' "iops" : 102324.525068,\n'
' "runtime" : 7001,\n'
' "total_ios" : 716374,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1543,\n'
' "max" : 18425,\n'
' "mean" : 1610.136684,\n'
' "stddev" : 115.405056,\n'
' "N" : 716374\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 400,\n'
' "max" : 1295729,\n'
' "mean" : 7923.244064,\n'
' "stddev" : '
'2451.556179,\n'
' "N" : 716374,\n'
' "percentile" : {\n'
' "1.000000" : 7392,\n'
' "5.000000" : 7456,\n'
' "10.000000" : 7456,\n'
' "20.000000" : 7520,\n'
' "30.000000" : 7520,\n'
' "40.000000" : 7584,\n'
' "50.000000" : 7648,\n'
' "60.000000" : 7648,\n'
' "70.000000" : 7712,\n'
' "80.000000" : 7712,\n'
' "90.000000" : 7776,\n'
' "95.000000" : 8160,\n'
' "99.000000" : 17280,\n'
' "99.500000" : 20096,\n'
' "99.900000" : 22656,\n'
' "99.950000" : 32384,\n'
' "99.990000" : 65280\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 8926,\n'
' "max" : 1297322,\n'
' "mean" : 9533.380748,\n'
' "stddev" : '
'2454.613923,\n'
' "N" : 716374\n'
' },\n'
' "bw_min" : 408576,\n'
' "bw_max" : 410869,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'409391.785714,\n'
' "bw_dev" : 583.681180,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 102144,\n'
' "iops_max" : 102717,\n'
' "iops_mean" : '
'102347.928571,\n'
' "iops_stddev" : '
'145.871632,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 6.957143,\n'
' "sys_cpu" : 93.000000,\n'
' "ctx" : 48,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.128419,\n'
' "20" : 3.324520,\n'
' "50" : 0.511325,\n'
' "100" : 0.034619,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=85097: Tue '
'Oct 25 23:27:12 2022\n'
' read: IOPS=102k, BW=400MiB/s '
'(419MB/s)(2798MiB/7001msec)\n'
' slat (nsec): min=1543, '
'max=18425, avg=1610.14, '
'stdev=115.41\n'
' clat (nsec): min=400, '
'max=1295.7k, avg=7923.24, '
'stdev=2451.56\n'
' lat (usec): min=8, max=1297, '
'avg= 9.53, stdev= 2.45\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7392], 5.00th=[ '
'7456], 10.00th=[ 7456], 20.00th=[ '
'7520],\n'
' | 30.00th=[ 7520], 40.00th=[ '
'7584], 50.00th=[ 7648], 60.00th=[ '
'7648],\n'
' | 70.00th=[ 7712], 80.00th=[ '
'7712], 90.00th=[ 7776], 95.00th=[ '
'8160],\n'
' | 99.00th=[17280], '
'99.50th=[20096], 99.90th=[22656], '
'99.95th=[32384],\n'
' | 99.99th=[65280]\n'
' bw ( KiB/s): min=408576, '
'max=410869, per=100.00%, '
'avg=409391.79, stdev=583.68, '
'samples=14\n'
' iops : min=102144, '
'max=102717, avg=102347.93, '
'stdev=145.87, samples=14\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.13%, 20=3.32%, 50=0.51%, '
'100=0.03%\n'
' lat (usec) : 250=0.01%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=6.96%, '
'sys=93.00%, ctx=48, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=716374,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=400MiB/s (419MB/s), '
'400MiB/s-400MiB/s '
'(419MB/s-419MB/s), io=2798MiB '
'(2934MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_51.output'),
'state': {'begin': 1666726142.863593,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7923583984375,
'end': 1666726143.6559515,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_51.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_51.state')},
'cmd_52': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_52.output'),
'state': {'begin': 1666726145.6664793,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7846190929412842,
'end': 1666726146.4510984,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_52.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_52.state')},
'cmd_53': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_53.output'),
'state': {'begin': 1666726146.4534276,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.453392505645752,
'end': 1666726157.90682,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_53.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_53.state')},
'cmd_54': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733247,\n'
' "timestamp_ms" : '
'1666733247334,\n'
' "time" : "Tue Oct 25 23:27:27 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2939764736,\n'
' "io_kbytes" : 2870864,\n'
' "bw_bytes" : 419906404,\n'
' "bw" : 410064,\n'
' "iops" : 102516.211970,\n'
' "runtime" : 7001,\n'
' "total_ios" : 717716,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1522,\n'
' "max" : 14347,\n'
' "mean" : 1578.864615,\n'
' "stddev" : 120.389006,\n'
' "N" : 717716\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 241,\n'
' "max" : 96561,\n'
' "mean" : 7938.478975,\n'
' "stddev" : '
'1946.448668,\n'
' "N" : 717716,\n'
' "percentile" : {\n'
' "1.000000" : 7456,\n'
' "5.000000" : 7456,\n'
' "10.000000" : 7456,\n'
' "20.000000" : 7520,\n'
' "30.000000" : 7520,\n'
' "40.000000" : 7584,\n'
' "50.000000" : 7648,\n'
' "60.000000" : 7648,\n'
' "70.000000" : 7712,\n'
' "80.000000" : 7712,\n'
' "90.000000" : 7776,\n'
' "95.000000" : 8160,\n'
' "99.000000" : 17280,\n'
' "99.500000" : 20352,\n'
' "99.900000" : 22912,\n'
' "99.950000" : 32640,\n'
' "99.990000" : 65280\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 8926,\n'
' "max" : 98124,\n'
' "mean" : 9517.343590,\n'
' "stddev" : '
'1950.790720,\n'
' "N" : 717716\n'
' },\n'
' "bw_min" : 409424,\n'
' "bw_max" : 411160,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'410161.928571,\n'
' "bw_dev" : 477.509318,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 102354,\n'
' "iops_max" : 102790,\n'
' "iops_mean" : '
'102540.428571,\n'
' "iops_stddev" : '
'119.741204,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 8.157143,\n'
' "sys_cpu" : 91.814286,\n'
' "ctx" : 5,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.122422,\n'
' "20" : 3.278456,\n'
' "50" : 0.559135,\n'
' "100" : 0.038316,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=85148: Tue '
'Oct 25 23:27:27 2022\n'
' read: IOPS=103k, BW=400MiB/s '
'(420MB/s)(2804MiB/7001msec)\n'
' slat (nsec): min=1522, '
'max=14347, avg=1578.86, '
'stdev=120.39\n'
' clat (nsec): min=241, '
'max=96561, avg=7938.48, '
'stdev=1946.45\n'
' lat (nsec): min=8926, '
'max=98124, avg=9517.34, '
'stdev=1950.79\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7456], 5.00th=[ '
'7456], 10.00th=[ 7456], 20.00th=[ '
'7520],\n'
' | 30.00th=[ 7520], 40.00th=[ '
'7584], 50.00th=[ 7648], 60.00th=[ '
'7648],\n'
' | 70.00th=[ 7712], 80.00th=[ '
'7712], 90.00th=[ 7776], 95.00th=[ '
'8160],\n'
' | 99.00th=[17280], '
'99.50th=[20352], 99.90th=[22912], '
'99.95th=[32640],\n'
' | 99.99th=[65280]\n'
' bw ( KiB/s): min=409424, '
'max=411160, per=100.00%, '
'avg=410161.93, stdev=477.51, '
'samples=14\n'
' iops : min=102354, '
'max=102790, avg=102540.43, '
'stdev=119.74, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%, 750=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.12%, 20=3.28%, 50=0.56%, '
'100=0.04%\n'
' cpu : usr=8.16%, '
'sys=91.81%, ctx=5, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=717716,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=400MiB/s (420MB/s), '
'400MiB/s-400MiB/s '
'(420MB/s-420MB/s), io=2804MiB '
'(2940MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_54.output'),
'state': {'begin': 1666726157.9090652,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7853574752807617,
'end': 1666726158.6944227,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_54.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_54.state')},
'cmd_55': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_55.output'),
'state': {'begin': 1666726159.6800706,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.782827615737915,
'end': 1666726160.4628983,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_55.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_55.state')},
'cmd_56': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_56.output'),
'state': {'begin': 1666726160.4650114,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring-xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.04825758934021,
'end': 1666726171.513269,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_56.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_56.state')},
'cmd_57': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733261,\n'
' "timestamp_ms" : '
'1666733261012,\n'
' "time" : "Tue Oct 25 23:27:41 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1795207168,\n'
' "io_kbytes" : 1753132,\n'
' "bw_bytes" : 256421535,\n'
' "bw" : 250411,\n'
' "iops" : 62602.913869,\n'
' "runtime" : 7001,\n'
' "total_ios" : 438283,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2655,\n'
' "max" : 27832,\n'
' "mean" : 2842.061771,\n'
' "stddev" : 230.543540,\n'
' "N" : 438283\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 281,\n'
' "max" : 122980,\n'
' "mean" : 12554.664060,\n'
' "stddev" : '
'2139.295770,\n'
' "N" : 438283,\n'
' "percentile" : {\n'
' "1.000000" : 11968,\n'
' "5.000000" : 12096,\n'
' "10.000000" : 12096,\n'
' "20.000000" : 12096,\n'
' "30.000000" : 12224,\n'
' "40.000000" : 12224,\n'
' "50.000000" : 12224,\n'
' "60.000000" : 12352,\n'
' "70.000000" : 12352,\n'
' "80.000000" : 12352,\n'
' "90.000000" : 12480,\n'
' "95.000000" : 12608,\n'
' "99.000000" : 22144,\n'
' "99.500000" : 28288,\n'
' "99.900000" : 40704,\n'
' "99.950000" : 54016,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12182,\n'
' "max" : 125816,\n'
' "mean" : 15396.725830,\n'
' "stddev" : '
'2157.279613,\n'
' "N" : 438283\n'
' },\n'
' "bw_min" : 249896,\n'
' "bw_max" : 251094,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'250470.142857,\n'
' "bw_dev" : 335.320250,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 62472,\n'
' "iops_max" : 62773,\n'
' "iops_mean" : '
'62617.500000,\n'
' "iops_stddev" : '
'83.901728,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 10.585714,\n'
' "sys_cpu" : 38.442857,\n'
' "ctx" : 438270,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 98.596113,\n'
' "50" : 1.331788,\n'
' "100" : 0.064798,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=85187: Tue '
'Oct 25 23:27:41 2022\n'
' read: IOPS=62.6k, BW=245MiB/s '
'(256MB/s)(1712MiB/7001msec)\n'
' slat (nsec): min=2655, '
'max=27832, avg=2842.06, '
'stdev=230.54\n'
' clat (nsec): min=281, '
'max=122980, avg=12554.66, '
'stdev=2139.30\n'
' lat (usec): min=12, max=125, '
'avg=15.40, stdev= 2.16\n'
' clat percentiles (nsec):\n'
' | 1.00th=[11968], '
'5.00th=[12096], 10.00th=[12096], '
'20.00th=[12096],\n'
' | 30.00th=[12224], '
'40.00th=[12224], 50.00th=[12224], '
'60.00th=[12352],\n'
' | 70.00th=[12352], '
'80.00th=[12352], 90.00th=[12480], '
'95.00th=[12608],\n'
' | 99.00th=[22144], '
'99.50th=[28288], 99.90th=[40704], '
'99.95th=[54016],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=249896, '
'max=251094, per=100.00%, '
'avg=250470.14, stdev=335.32, '
'samples=14\n'
' iops : min=62472, '
'max=62773, avg=62617.50, '
'stdev=83.90, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=98.60%, 50=1.33%, '
'100=0.06%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=10.59%, '
'sys=38.44%, ctx=438270, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=438283,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=245MiB/s (256MB/s), '
'245MiB/s-245MiB/s '
'(256MB/s-256MB/s), io=1712MiB '
'(1795MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_57.output'),
'state': {'begin': 1666726171.5161881,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7912197113037109,
'end': 1666726172.3074079,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_57.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_57.state')},
'cmd_58': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_58.output'),
'state': {'begin': 1666726173.3039281,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7914633750915527,
'end': 1666726174.0953915,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_58.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_58.state')},
'cmd_59': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_59.output'),
'state': {'begin': 1666726174.097773,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring-xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.052335023880005,
'end': 1666726185.150108,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_59.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_59.state')},
'cmd_60': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733274,\n'
' "timestamp_ms" : '
'1666733274648,\n'
' "time" : "Tue Oct 25 23:27:54 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1804898304,\n'
' "io_kbytes" : 1762596,\n'
' "bw_bytes" : 257805785,\n'
' "bw" : 251763,\n'
' "iops" : 62940.865591,\n'
' "runtime" : 7001,\n'
' "total_ios" : 440649,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2645,\n'
' "max" : 27261,\n'
' "mean" : 2816.685162,\n'
' "stddev" : 258.194088,\n'
' "N" : 440649\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 380,\n'
' "max" : 467586,\n'
' "mean" : 12522.016850,\n'
' "stddev" : '
'2175.039916,\n'
' "N" : 440649,\n'
' "percentile" : {\n'
' "1.000000" : 11968,\n'
' "5.000000" : 12096,\n'
' "10.000000" : 12096,\n'
' "20.000000" : 12096,\n'
' "30.000000" : 12224,\n'
' "40.000000" : 12224,\n'
' "50.000000" : 12224,\n'
' "60.000000" : 12352,\n'
' "70.000000" : 12352,\n'
' "80.000000" : 12352,\n'
' "90.000000" : 12480,\n'
' "95.000000" : 12608,\n'
' "99.000000" : 22144,\n'
' "99.500000" : 28032,\n'
' "99.900000" : 33536,\n'
' "99.950000" : 50432,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12243,\n'
' "max" : 470382,\n'
' "mean" : 15338.702012,\n'
' "stddev" : '
'2192.213103,\n'
' "N" : 440649\n'
' },\n'
' "bw_min" : 250829,\n'
' "bw_max" : 252600,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'251826.071429,\n'
' "bw_dev" : 417.064921,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 62707,\n'
' "iops_max" : 63150,\n'
' "iops_mean" : '
'62956.500000,\n'
' "iops_stddev" : '
'104.312216,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 11.042857,\n'
' "sys_cpu" : 38.328571,\n'
' "ctx" : 440630,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 98.638372,\n'
' "50" : 1.295589,\n'
' "100" : 0.056054,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=85227: Tue '
'Oct 25 23:27:54 2022\n'
' read: IOPS=62.9k, BW=246MiB/s '
'(258MB/s)(1721MiB/7001msec)\n'
' slat (nsec): min=2645, '
'max=27261, avg=2816.69, '
'stdev=258.19\n'
' clat (nsec): min=380, '
'max=467586, avg=12522.02, '
'stdev=2175.04\n'
' lat (usec): min=12, max=470, '
'avg=15.34, stdev= 2.19\n'
' clat percentiles (nsec):\n'
' | 1.00th=[11968], '
'5.00th=[12096], 10.00th=[12096], '
'20.00th=[12096],\n'
' | 30.00th=[12224], '
'40.00th=[12224], 50.00th=[12224], '
'60.00th=[12352],\n'
' | 70.00th=[12352], '
'80.00th=[12352], 90.00th=[12480], '
'95.00th=[12608],\n'
' | 99.00th=[22144], '
'99.50th=[28032], 99.90th=[33536], '
'99.95th=[50432],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=250829, '
'max=252600, per=100.00%, '
'avg=251826.07, stdev=417.06, '
'samples=14\n'
' iops : min=62707, '
'max=63150, avg=62956.50, '
'stdev=104.31, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=98.64%, 50=1.30%, '
'100=0.06%\n'
' lat (usec) : 250=0.01%, '
'500=0.01%\n'
' cpu : usr=11.04%, '
'sys=38.33%, ctx=440630, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=440649,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=246MiB/s (258MB/s), '
'246MiB/s-246MiB/s '
'(258MB/s-258MB/s), io=1721MiB '
'(1805MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_60.output'),
'state': {'begin': 1666726185.1532598,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7865962982177734,
'end': 1666726185.939856,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_60.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_60.state')},
'cmd_61': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_io_uring-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_61.output'),
'state': {'begin': 1666726186.9379005,
'cmd': 'rm '
'/tmp/fio-output_4K_1_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7873010635375977,
'end': 1666726187.7252016,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_61.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_61.state')},
'cmd_62': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_62.output'),
'state': {'begin': 1666726187.7272713,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_io_uring-xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.057294368743896,
'end': 1666726198.7845657,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_62.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_62.state')},
'cmd_63': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733288,\n'
' "timestamp_ms" : '
'1666733288284,\n'
' "time" : "Tue Oct 25 23:28:08 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1813983232,\n'
' "io_kbytes" : 1771468,\n'
' "bw_bytes" : 259103446,\n'
' "bw" : 253030,\n'
' "iops" : 63257.534638,\n'
' "runtime" : 7001,\n'
' "total_ios" : 442866,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2584,\n'
' "max" : 28093,\n'
' "mean" : 2753.253893,\n'
' "stddev" : 286.498944,\n'
' "N" : 442867\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 421,\n'
' "max" : 181340,\n'
' "mean" : 12481.790944,\n'
' "stddev" : '
'2028.063473,\n'
' "N" : 442866,\n'
' "percentile" : {\n'
' "1.000000" : 11968,\n'
' "5.000000" : 11968,\n'
' "10.000000" : 11968,\n'
' "20.000000" : 12096,\n'
' "30.000000" : 12096,\n'
' "40.000000" : 12224,\n'
' "50.000000" : 12224,\n'
' "60.000000" : 12224,\n'
' "70.000000" : 12352,\n'
' "80.000000" : 12352,\n'
' "90.000000" : 12352,\n'
' "95.000000" : 12608,\n'
' "99.000000" : 22144,\n'
' "99.500000" : 27776,\n'
' "99.900000" : 32384,\n'
' "99.950000" : 50432,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12102,\n'
' "max" : 184025,\n'
' "mean" : 15235.045014,\n'
' "stddev" : '
'2052.149286,\n'
' "N" : 442866\n'
' },\n'
' "bw_min" : 252120,\n'
' "bw_max" : 254468,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'253092.857143,\n'
' "bw_dev" : 627.819040,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 63030,\n'
' "iops_max" : 63617,\n'
' "iops_mean" : '
'63273.214286,\n'
' "iops_stddev" : '
'156.954760,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 10.757143,\n'
' "sys_cpu" : 38.157143,\n'
' "ctx" : 442837,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 98.632995,\n'
' "50" : 1.304909,\n'
' "100" : 0.052612,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=85266: Tue '
'Oct 25 23:28:08 2022\n'
' read: IOPS=63.3k, BW=247MiB/s '
'(259MB/s)(1730MiB/7001msec)\n'
' slat (nsec): min=2584, '
'max=28093, avg=2753.25, '
'stdev=286.50\n'
' clat (nsec): min=421, '
'max=181340, avg=12481.79, '
'stdev=2028.06\n'
' lat (usec): min=12, max=184, '
'avg=15.24, stdev= 2.05\n'
' clat percentiles (nsec):\n'
' | 1.00th=[11968], '
'5.00th=[11968], 10.00th=[11968], '
'20.00th=[12096],\n'
' | 30.00th=[12096], '
'40.00th=[12224], 50.00th=[12224], '
'60.00th=[12224],\n'
' | 70.00th=[12352], '
'80.00th=[12352], 90.00th=[12352], '
'95.00th=[12608],\n'
' | 99.00th=[22144], '
'99.50th=[27776], 99.90th=[32384], '
'99.95th=[50432],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=252120, '
'max=254468, per=100.00%, '
'avg=253092.86, stdev=627.82, '
'samples=14\n'
' iops : min=63030, '
'max=63617, avg=63273.21, '
'stdev=156.95, samples=14\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%\n'
' lat (usec) : 10=0.01%, '
'20=98.63%, 50=1.30%, 100=0.05%, '
'250=0.01%\n'
' cpu : usr=10.76%, '
'sys=38.16%, ctx=442837, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=442866,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=247MiB/s (259MB/s), '
'247MiB/s-247MiB/s '
'(259MB/s-259MB/s), io=1730MiB '
'(1814MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_63.output'),
'state': {'begin': 1666726198.7879772,
'cmd': 'cat '
'/tmp/fio-output_4K_1_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7874362468719482,
'end': 1666726199.5754135,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_63.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_63.state')},
'cmd_64': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_libaio-reference_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_64.output'),
'state': {'begin': 1666726200.562892,
'cmd': 'rm '
'/tmp/fio-output_4K_1_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7885828018188477,
'end': 1666726201.3514748,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_64.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_64.state')},
'cmd_65': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_65.output'),
'state': {'begin': 1666726201.3534098,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_libaio-reference_0.txt"',
'cwd': 'None',
'elapsed': 11.085954666137695,
'end': 1666726212.4393644,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_65.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_65.state')},
'cmd_66': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=libaio, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733301,\n'
' "timestamp_ms" : '
'1666733301938,\n'
' "time" : "Tue Oct 25 23:28:21 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1673572352,\n'
' "io_kbytes" : 1634348,\n'
' "bw_bytes" : 239047614,\n'
' "bw" : 233444,\n'
' "iops" : 58361.234109,\n'
' "runtime" : 7001,\n'
' "total_ios" : 408587,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2836,\n'
' "max" : 17292,\n'
' "mean" : 3074.658611,\n'
' "stddev" : 158.340253,\n'
' "N" : 408587\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 802,\n'
' "max" : 118943,\n'
' "mean" : 13512.992719,\n'
' "stddev" : '
'2095.582789,\n'
' "N" : 408587,\n'
' "percentile" : {\n'
' "1.000000" : 12864,\n'
' "5.000000" : 12992,\n'
' "10.000000" : 13120,\n'
' "20.000000" : 13120,\n'
' "30.000000" : 13248,\n'
' "40.000000" : 13248,\n'
' "50.000000" : 13248,\n'
' "60.000000" : 13248,\n'
' "70.000000" : 13248,\n'
' "80.000000" : 13376,\n'
' "90.000000" : 13376,\n'
' "95.000000" : 13504,\n'
' "99.000000" : 23680,\n'
' "99.500000" : 28800,\n'
' "99.900000" : 33536,\n'
' "99.950000" : 50432,\n'
' "99.990000" : 71168\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 14397,\n'
' "max" : 122139,\n'
' "mean" : 16587.651330,\n'
' "stddev" : '
'2102.634447,\n'
' "N" : 408587\n'
' },\n'
' "bw_min" : 233168,\n'
' "bw_max" : 233976,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'233596.714286,\n'
' "bw_dev" : 275.558574,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 58292,\n'
' "iops_max" : 58494,\n'
' "iops_mean" : '
'58399.000000,\n'
' "iops_stddev" : '
'68.835367,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 9.100000,\n'
' "sys_cpu" : 32.528571,\n'
' "ctx" : 408588,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 98.564810,\n'
' "50" : 1.376696,\n'
' "100" : 0.051641,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 586804,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 5712,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 5712,\n'
' "util" : 99.108381\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=85307: Tue '
'Oct 25 23:28:21 2022\n'
' read: IOPS=58.4k, BW=228MiB/s '
'(239MB/s)(1596MiB/7001msec)\n'
' slat (nsec): min=2836, '
'max=17292, avg=3074.66, '
'stdev=158.34\n'
' clat (nsec): min=802, '
'max=118943, avg=13512.99, '
'stdev=2095.58\n'
' lat (usec): min=14, max=122, '
'avg=16.59, stdev= 2.10\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12864], '
'5.00th=[12992], 10.00th=[13120], '
'20.00th=[13120],\n'
' | 30.00th=[13248], '
'40.00th=[13248], 50.00th=[13248], '
'60.00th=[13248],\n'
' | 70.00th=[13248], '
'80.00th=[13376], 90.00th=[13376], '
'95.00th=[13504],\n'
' | 99.00th=[23680], '
'99.50th=[28800], 99.90th=[33536], '
'99.95th=[50432],\n'
' | 99.99th=[71168]\n'
' bw ( KiB/s): min=233168, '
'max=233976, per=100.00%, '
'avg=233596.71, stdev=275.56, '
'samples=14\n'
' iops : min=58292, '
'max=58494, avg=58399.00, '
'stdev=68.84, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 10=0.01%, '
'20=98.56%, 50=1.38%, 100=0.05%, '
'250=0.01%\n'
' cpu : usr=9.10%, '
'sys=32.53%, ctx=408588, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=408587,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=228MiB/s (239MB/s), '
'228MiB/s-228MiB/s '
'(239MB/s-239MB/s), io=1596MiB '
'(1674MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=586804/0, '
'merge=0/0, ticks=5712/0, '
'in_queue=5712, util=99.11%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_66.output'),
'state': {'begin': 1666726212.4419904,
'cmd': 'cat '
'/tmp/fio-output_4K_1_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7871720790863037,
'end': 1666726213.2291625,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_66.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_66.state')},
'cmd_67': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_libaio-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_67.output'),
'state': {'begin': 1666726214.2225513,
'cmd': 'rm '
'/tmp/fio-output_4K_1_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7865972518920898,
'end': 1666726215.0091486,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_67.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_67.state')},
'cmd_68': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_68.output'),
'state': {'begin': 1666726215.0111077,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_libaio-reference_1.txt"',
'cwd': 'None',
'elapsed': 11.077730178833008,
'end': 1666726226.0888379,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_68.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_68.state')},
'cmd_69': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=libaio, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733315,\n'
' "timestamp_ms" : '
'1666733315587,\n'
' "time" : "Tue Oct 25 23:28:35 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1801781248,\n'
' "io_kbytes" : 1759552,\n'
' "bw_bytes" : 257360555,\n'
' "bw" : 251328,\n'
' "iops" : 62832.166833,\n'
' "runtime" : 7001,\n'
' "total_ios" : 439888,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2765,\n'
' "max" : 26921,\n'
' "mean" : 2932.077167,\n'
' "stddev" : 192.262685,\n'
' "N" : 439888\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 881,\n'
' "max" : 77566,\n'
' "mean" : 12358.918272,\n'
' "stddev" : '
'1970.343765,\n'
' "N" : 439888,\n'
' "percentile" : {\n'
' "1.000000" : 11840,\n'
' "5.000000" : 11840,\n'
' "10.000000" : 11968,\n'
' "20.000000" : 11968,\n'
' "30.000000" : 11968,\n'
' "40.000000" : 12096,\n'
' "50.000000" : 12096,\n'
' "60.000000" : 12096,\n'
' "70.000000" : 12224,\n'
' "80.000000" : 12224,\n'
' "90.000000" : 12352,\n'
' "95.000000" : 12480,\n'
' "99.000000" : 21888,\n'
' "99.500000" : 27264,\n'
' "99.900000" : 31616,\n'
' "99.950000" : 49920,\n'
' "99.990000" : 67072\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12002,\n'
' "max" : 80582,\n'
' "mean" : 15290.995440,\n'
' "stddev" : '
'1982.252033,\n'
' "N" : 439888\n'
' },\n'
' "bw_min" : 250264,\n'
' "bw_max" : 252440,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'251431.928571,\n'
' "bw_dev" : 631.720540,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 62566,\n'
' "iops_max" : 63110,\n'
' "iops_mean" : '
'62857.928571,\n'
' "iops_stddev" : '
'157.898928,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 11.057143,\n'
' "sys_cpu" : 37.400000,\n'
' "ctx" : 439880,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 98.638062,\n'
' "50" : 1.307151,\n'
' "100" : 0.047057,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 631256,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 5943,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 5943,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=85345: Tue '
'Oct 25 23:28:35 2022\n'
' read: IOPS=62.8k, BW=245MiB/s '
'(257MB/s)(1718MiB/7001msec)\n'
' slat (nsec): min=2765, '
'max=26921, avg=2932.08, '
'stdev=192.26\n'
' clat (nsec): min=881, '
'max=77566, avg=12358.92, '
'stdev=1970.34\n'
' lat (nsec): min=12002, '
'max=80582, avg=15291.00, '
'stdev=1982.25\n'
' clat percentiles (nsec):\n'
' | 1.00th=[11840], '
'5.00th=[11840], 10.00th=[11968], '
'20.00th=[11968],\n'
' | 30.00th=[11968], '
'40.00th=[12096], 50.00th=[12096], '
'60.00th=[12096],\n'
' | 70.00th=[12224], '
'80.00th=[12224], 90.00th=[12352], '
'95.00th=[12480],\n'
' | 99.00th=[21888], '
'99.50th=[27264], 99.90th=[31616], '
'99.95th=[49920],\n'
' | 99.99th=[67072]\n'
' bw ( KiB/s): min=250264, '
'max=252440, per=100.00%, '
'avg=251431.93, stdev=631.72, '
'samples=14\n'
' iops : min=62566, '
'max=63110, avg=62857.93, '
'stdev=157.90, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'10=0.01%, 20=98.64%, 50=1.31%, '
'100=0.05%\n'
' cpu : usr=11.06%, '
'sys=37.40%, ctx=439880, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=439888,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=245MiB/s (257MB/s), '
'245MiB/s-245MiB/s '
'(257MB/s-257MB/s), io=1718MiB '
'(1802MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=631256/0, '
'merge=0/0, ticks=5943/0, '
'in_queue=5943, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_69.output'),
'state': {'begin': 1666726226.0914607,
'cmd': 'cat '
'/tmp/fio-output_4K_1_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7961251735687256,
'end': 1666726226.8875859,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_69.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_69.state')},
'cmd_70': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_libaio-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_70.output'),
'state': {'begin': 1666726227.883852,
'cmd': 'rm '
'/tmp/fio-output_4K_1_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7791063785552979,
'end': 1666726228.6629584,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_70.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_70.state')},
'cmd_71': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_71.output'),
'state': {'begin': 1666726228.665182,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_libaio-reference_2.txt"',
'cwd': 'None',
'elapsed': 11.08476448059082,
'end': 1666726239.7499466,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_71.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_71.state')},
'cmd_72': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=libaio, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733329,\n'
' "timestamp_ms" : '
'1666733329246,\n'
' "time" : "Tue Oct 25 23:28:49 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1831936000,\n'
' "io_kbytes" : 1789000,\n'
' "bw_bytes" : 261667761,\n'
' "bw" : 255534,\n'
' "iops" : 63883.588059,\n'
' "runtime" : 7001,\n'
' "total_ios" : 447249,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2605,\n'
' "max" : 37320,\n'
' "mean" : 2851.803557,\n'
' "stddev" : 243.670282,\n'
' "N" : 447250\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 872,\n'
' "max" : 136185,\n'
' "mean" : 12256.182628,\n'
' "stddev" : '
'1988.047396,\n'
' "N" : 447249,\n'
' "percentile" : {\n'
' "1.000000" : 11712,\n'
' "5.000000" : 11712,\n'
' "10.000000" : 11840,\n'
' "20.000000" : 11840,\n'
' "30.000000" : 11840,\n'
' "40.000000" : 11968,\n'
' "50.000000" : 11968,\n'
' "60.000000" : 12096,\n'
' "70.000000" : 12096,\n'
' "80.000000" : 12096,\n'
' "90.000000" : 12224,\n'
' "95.000000" : 12352,\n'
' "99.000000" : 22144,\n'
' "99.500000" : 26752,\n'
' "99.900000" : 31616,\n'
' "99.950000" : 49408,\n'
' "99.990000" : 63744\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12163,\n'
' "max" : 138971,\n'
' "mean" : 15107.986178,\n'
' "stddev" : '
'2008.944842,\n'
' "N" : 447249\n'
' },\n'
' "bw_min" : 254784,\n'
' "bw_max" : 257032,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'255632.357143,\n'
' "bw_dev" : 623.778265,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 63696,\n'
' "iops_max" : 64258,\n'
' "iops_mean" : '
'63908.000000,\n'
' "iops_stddev" : '
'155.974850,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 10.242857,\n'
' "sys_cpu" : 37.271429,\n'
' "ctx" : 447233,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 98.553155,\n'
' "50" : 1.395867,\n'
' "100" : 0.044047,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 640196,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 6020,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 6020,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=85383: Tue '
'Oct 25 23:28:49 2022\n'
' read: IOPS=63.9k, BW=250MiB/s '
'(262MB/s)(1747MiB/7001msec)\n'
' slat (nsec): min=2605, '
'max=37320, avg=2851.80, '
'stdev=243.67\n'
' clat (nsec): min=872, '
'max=136185, avg=12256.18, '
'stdev=1988.05\n'
' lat (usec): min=12, max=138, '
'avg=15.11, stdev= 2.01\n'
' clat percentiles (nsec):\n'
' | 1.00th=[11712], '
'5.00th=[11712], 10.00th=[11840], '
'20.00th=[11840],\n'
' | 30.00th=[11840], '
'40.00th=[11968], 50.00th=[11968], '
'60.00th=[12096],\n'
' | 70.00th=[12096], '
'80.00th=[12096], 90.00th=[12224], '
'95.00th=[12352],\n'
' | 99.00th=[22144], '
'99.50th=[26752], 99.90th=[31616], '
'99.95th=[49408],\n'
' | 99.99th=[63744]\n'
' bw ( KiB/s): min=254784, '
'max=257032, per=100.00%, '
'avg=255632.36, stdev=623.78, '
'samples=14\n'
' iops : min=63696, '
'max=64258, avg=63908.00, '
'stdev=155.97, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.01%, 20=98.55%, '
'50=1.40%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=10.24%, '
'sys=37.27%, ctx=447233, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=447249,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=250MiB/s (262MB/s), '
'250MiB/s-250MiB/s '
'(262MB/s-262MB/s), io=1747MiB '
'(1832MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=640196/0, '
'merge=0/0, ticks=6020/0, '
'in_queue=6020, util=99.09%\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_72.output'),
'state': {'begin': 1666726239.75269,
'cmd': 'cat '
'/tmp/fio-output_4K_1_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7971940040588379,
'end': 1666726240.549884,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_72.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_72.state')},
'cmd_73': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_libaio-bdev_aio_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_73.output'),
'state': {'begin': 1666726242.5884023,
'cmd': 'rm '
'/tmp/fio-output_4K_1_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.7865762710571289,
'end': 1666726243.3749785,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_73.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_73.state')},
'cmd_74': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_74.output'),
'state': {'begin': 1666726243.3770688,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_libaio-bdev_aio_0.txt"',
'cwd': 'None',
'elapsed': 11.566319227218628,
'end': 1666726254.943388,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_74.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_74.state')},
'cmd_75': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733344,\n'
' "timestamp_ms" : '
'1666733344307,\n'
' "time" : "Tue Oct 25 23:29:04 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2908217344,\n'
' "io_kbytes" : 2840056,\n'
' "bw_bytes" : 415400277,\n'
' "bw" : 405664,\n'
' "iops" : 101416.083417,\n'
' "runtime" : 7001,\n'
' "total_ios" : 710014,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1562,\n'
' "max" : 34896,\n'
' "mean" : 1634.305671,\n'
' "stddev" : 120.389758,\n'
' "N" : 710014\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 221,\n'
' "max" : 113843,\n'
' "mean" : 7995.203314,\n'
' "stddev" : '
'1932.985662,\n'
' "N" : 710014,\n'
' "percentile" : {\n'
' "1.000000" : 7520,\n'
' "5.000000" : 7520,\n'
' "10.000000" : 7520,\n'
' "20.000000" : 7584,\n'
' "30.000000" : 7584,\n'
' "40.000000" : 7648,\n'
' "50.000000" : 7712,\n'
' "60.000000" : 7712,\n'
' "70.000000" : 7776,\n'
' "80.000000" : 7840,\n'
' "90.000000" : 7840,\n'
' "95.000000" : 8384,\n'
' "99.000000" : 17280,\n'
' "99.500000" : 19840,\n'
' "99.900000" : 23680,\n'
' "99.950000" : 40704,\n'
' "99.990000" : 65280\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9027,\n'
' "max" : 115476,\n'
' "mean" : 9629.508986,\n'
' "stddev" : '
'1936.932387,\n'
' "N" : 710014\n'
' },\n'
' "bw_min" : 404904,\n'
' "bw_max" : 407032,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'405763.785714,\n'
' "bw_dev" : 726.004893,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 101226,\n'
' "iops_max" : 101758,\n'
' "iops_mean" : '
'101440.928571,\n'
' "iops_stddev" : '
'181.477044,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 85.300000,\n'
' "sys_cpu" : 14.671429,\n'
' "ctx" : 18,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.102471,\n'
' "20" : 3.421904,\n'
' "50" : 0.436611,\n'
' "100" : 0.037605,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=85432: Tue '
'Oct 25 23:29:04 2022\n'
' read: IOPS=101k, BW=396MiB/s '
'(415MB/s)(2773MiB/7001msec)\n'
' slat (nsec): min=1562, '
'max=34896, avg=1634.31, '
'stdev=120.39\n'
' clat (nsec): min=221, '
'max=113843, avg=7995.20, '
'stdev=1932.99\n'
' lat (usec): min=9, max=115, '
'avg= 9.63, stdev= 1.94\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7520], 5.00th=[ '
'7520], 10.00th=[ 7520], 20.00th=[ '
'7584],\n'
' | 30.00th=[ 7584], 40.00th=[ '
'7648], 50.00th=[ 7712], 60.00th=[ '
'7712],\n'
' | 70.00th=[ 7776], 80.00th=[ '
'7840], 90.00th=[ 7840], 95.00th=[ '
'8384],\n'
' | 99.00th=[17280], '
'99.50th=[19840], 99.90th=[23680], '
'99.95th=[40704],\n'
' | 99.99th=[65280]\n'
' bw ( KiB/s): min=404904, '
'max=407032, per=100.00%, '
'avg=405763.79, stdev=726.00, '
'samples=14\n'
' iops : min=101226, '
'max=101758, avg=101440.93, '
'stdev=181.48, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 10=96.10%, '
'20=3.42%, 50=0.44%, 100=0.04%, '
'250=0.01%\n'
' cpu : usr=85.30%, '
'sys=14.67%, ctx=18, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=710014,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=396MiB/s (415MB/s), '
'396MiB/s-396MiB/s '
'(415MB/s-415MB/s), io=2773MiB '
'(2908MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_75.output'),
'state': {'begin': 1666726254.9463606,
'cmd': 'cat '
'/tmp/fio-output_4K_1_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.7987728118896484,
'end': 1666726255.7451334,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_75.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_75.state')},
'cmd_76': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_libaio-bdev_aio_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_76.output'),
'state': {'begin': 1666726257.8994217,
'cmd': 'rm '
'/tmp/fio-output_4K_1_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7967190742492676,
'end': 1666726258.6961408,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_76.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_76.state')},
'cmd_77': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_77.output'),
'state': {'begin': 1666726258.698414,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_libaio-bdev_aio_1.txt"',
'cwd': 'None',
'elapsed': 11.520707607269287,
'end': 1666726270.2191217,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_77.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_77.state')},
'cmd_78': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733359,\n'
' "timestamp_ms" : '
'1666733359618,\n'
' "time" : "Tue Oct 25 23:29:19 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2864140288,\n'
' "io_kbytes" : 2797012,\n'
' "bw_bytes" : 409104454,\n'
' "bw" : 399516,\n'
' "iops" : 99879.017283,\n'
' "runtime" : 7001,\n'
' "total_ios" : 699253,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1643,\n'
' "max" : 27060,\n'
' "mean" : 1697.425543,\n'
' "stddev" : 169.441086,\n'
' "N" : 699253\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 210,\n'
' "max" : 77946,\n'
' "mean" : 8068.355484,\n'
' "stddev" : '
'1872.173034,\n'
' "N" : 699253,\n'
' "percentile" : {\n'
' "1.000000" : 7648,\n'
' "5.000000" : 7712,\n'
' "10.000000" : 7712,\n'
' "20.000000" : 7712,\n'
' "30.000000" : 7712,\n'
' "40.000000" : 7712,\n'
' "50.000000" : 7776,\n'
' "60.000000" : 7776,\n'
' "70.000000" : 7776,\n'
' "80.000000" : 7776,\n'
' "90.000000" : 7904,\n'
' "95.000000" : 8512,\n'
' "99.000000" : 17280,\n'
' "99.500000" : 19328,\n'
' "99.900000" : 23936,\n'
' "99.950000" : 36608,\n'
' "99.990000" : 64768\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9247,\n'
' "max" : 79629,\n'
' "mean" : 9765.781026,\n'
' "stddev" : '
'1879.123838,\n'
' "N" : 699253\n'
' },\n'
' "bw_min" : 398744,\n'
' "bw_max" : 400280,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'399610.785714,\n'
' "bw_dev" : 354.371856,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 99686,\n'
' "iops_max" : 100070,\n'
' "iops_mean" : '
'99902.642857,\n'
' "iops_stddev" : '
'88.578177,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 85.242857,\n'
' "sys_cpu" : 14.728571,\n'
' "ctx" : 12,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.071522,\n'
' "20" : 3.639741,\n'
' "50" : 0.253127,\n'
' "100" : 0.032034,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=85481: Tue '
'Oct 25 23:29:19 2022\n'
' read: IOPS=99.9k, BW=390MiB/s '
'(409MB/s)(2731MiB/7001msec)\n'
' slat (nsec): min=1643, '
'max=27060, avg=1697.43, '
'stdev=169.44\n'
' clat (nsec): min=210, '
'max=77946, avg=8068.36, '
'stdev=1872.17\n'
' lat (nsec): min=9247, '
'max=79629, avg=9765.78, '
'stdev=1879.12\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7648], 5.00th=[ '
'7712], 10.00th=[ 7712], 20.00th=[ '
'7712],\n'
' | 30.00th=[ 7712], 40.00th=[ '
'7712], 50.00th=[ 7776], 60.00th=[ '
'7776],\n'
' | 70.00th=[ 7776], 80.00th=[ '
'7776], 90.00th=[ 7904], 95.00th=[ '
'8512],\n'
' | 99.00th=[17280], '
'99.50th=[19328], 99.90th=[23936], '
'99.95th=[36608],\n'
' | 99.99th=[64768]\n'
' bw ( KiB/s): min=398744, '
'max=400280, per=100.00%, '
'avg=399610.79, stdev=354.37, '
'samples=14\n'
' iops : min=99686, '
'max=100070, avg=99902.64, '
'stdev=88.58, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 10=96.07%, '
'20=3.64%, 50=0.25%, 100=0.03%\n'
' cpu : usr=85.24%, '
'sys=14.73%, ctx=12, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=699253,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=390MiB/s (409MB/s), '
'390MiB/s-390MiB/s '
'(409MB/s-409MB/s), io=2731MiB '
'(2864MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_78.output'),
'state': {'begin': 1666726270.2221115,
'cmd': 'cat '
'/tmp/fio-output_4K_1_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7994446754455566,
'end': 1666726271.0215561,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_78.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_78.state')},
'cmd_79': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_libaio-bdev_aio_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_79.output'),
'state': {'begin': 1666726273.0808809,
'cmd': 'rm '
'/tmp/fio-output_4K_1_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.7926232814788818,
'end': 1666726273.8735042,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_79.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_79.state')},
'cmd_80': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_80.output'),
'state': {'begin': 1666726273.8753877,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_libaio-bdev_aio_2.txt"',
'cwd': 'None',
'elapsed': 11.423839807510376,
'end': 1666726285.2992275,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_80.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_80.state')},
'cmd_81': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733374,\n'
' "timestamp_ms" : '
'1666733374688,\n'
' "time" : "Tue Oct 25 23:29:34 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2917703680,\n'
' "io_kbytes" : 2849320,\n'
' "bw_bytes" : 416814811,\n'
' "bw" : 407045,\n'
' "iops" : 101761.428571,\n'
' "runtime" : 7000,\n'
' "total_ios" : 712330,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1533,\n'
' "max" : 13756,\n'
' "mean" : 1593.976842,\n'
' "stddev" : 115.555755,\n'
' "N" : 712330\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 221,\n'
' "max" : 75782,\n'
' "mean" : 7996.381760,\n'
' "stddev" : '
'1896.986393,\n'
' "N" : 712330,\n'
' "percentile" : {\n'
' "1.000000" : 7520,\n'
' "5.000000" : 7520,\n'
' "10.000000" : 7520,\n'
' "20.000000" : 7584,\n'
' "30.000000" : 7584,\n'
' "40.000000" : 7648,\n'
' "50.000000" : 7712,\n'
' "60.000000" : 7712,\n'
' "70.000000" : 7776,\n'
' "80.000000" : 7840,\n'
' "90.000000" : 7840,\n'
' "95.000000" : 8256,\n'
' "99.000000" : 17280,\n'
' "99.500000" : 20096,\n'
' "99.900000" : 22656,\n'
' "99.950000" : 32384,\n'
' "99.990000" : 65280\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9007,\n'
' "max" : 77365,\n'
' "mean" : 9590.358602,\n'
' "stddev" : '
'1900.929895,\n'
' "N" : 712330\n'
' },\n'
' "bw_min" : 406344,\n'
' "bw_max" : 407800,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'407090.071429,\n'
' "bw_dev" : 420.417094,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 101586,\n'
' "iops_max" : 101950,\n'
' "iops_mean" : '
'101772.500000,\n'
' "iops_stddev" : '
'105.118797,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 85.871429,\n'
' "sys_cpu" : 14.100000,\n'
' "ctx" : 10,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.124268,\n'
' "20" : 3.367821,\n'
' "50" : 0.472955,\n'
' "100" : 0.033692,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=85531: Tue '
'Oct 25 23:29:34 2022\n'
' read: IOPS=102k, BW=398MiB/s '
'(417MB/s)(2783MiB/7000msec)\n'
' slat (nsec): min=1533, '
'max=13756, avg=1593.98, '
'stdev=115.56\n'
' clat (nsec): min=221, '
'max=75782, avg=7996.38, '
'stdev=1896.99\n'
' lat (nsec): min=9007, '
'max=77365, avg=9590.36, '
'stdev=1900.93\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7520], 5.00th=[ '
'7520], 10.00th=[ 7520], 20.00th=[ '
'7584],\n'
' | 30.00th=[ 7584], 40.00th=[ '
'7648], 50.00th=[ 7712], 60.00th=[ '
'7712],\n'
' | 70.00th=[ 7776], 80.00th=[ '
'7840], 90.00th=[ 7840], 95.00th=[ '
'8256],\n'
' | 99.00th=[17280], '
'99.50th=[20096], 99.90th=[22656], '
'99.95th=[32384],\n'
' | 99.99th=[65280]\n'
' bw ( KiB/s): min=406344, '
'max=407800, per=100.00%, '
'avg=407090.07, stdev=420.42, '
'samples=14\n'
' iops : min=101586, '
'max=101950, avg=101772.50, '
'stdev=105.12, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.12%, 20=3.37%, 50=0.47%, '
'100=0.03%\n'
' cpu : usr=85.87%, '
'sys=14.10%, ctx=10, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=712330,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=398MiB/s (417MB/s), '
'398MiB/s-398MiB/s '
'(417MB/s-417MB/s), io=2783MiB '
'(2918MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_81.output'),
'state': {'begin': 1666726285.3017569,
'cmd': 'cat '
'/tmp/fio-output_4K_1_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.8237886428833008,
'end': 1666726286.1255455,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_81.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_81.state')},
'cmd_82': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_82.output'),
'state': {'begin': 1666726288.2137384,
'cmd': 'rm '
'/tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7974402904510498,
'end': 1666726289.0111787,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_82.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_82.state')},
'cmd_83': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_83.output'),
'state': {'begin': 1666726289.013351,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.530867338180542,
'end': 1666726300.5442183,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_83.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_83.state')},
'cmd_84': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733389,\n'
' "timestamp_ms" : '
'1666733389939,\n'
' "time" : "Tue Oct 25 23:29:49 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2542465024,\n'
' "io_kbytes" : 2482876,\n'
' "bw_bytes" : 363157409,\n'
' "bw" : 354645,\n'
' "iops" : 88661.476932,\n'
' "runtime" : 7001,\n'
' "total_ios" : 620719,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1783,\n'
' "max" : 14928,\n'
' "mean" : 1926.784324,\n'
' "stddev" : 147.911807,\n'
' "N" : 620719\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 611,\n'
' "max" : 95610,\n'
' "mean" : 9103.494547,\n'
' "stddev" : '
'2144.724722,\n'
' "N" : 620719,\n'
' "percentile" : {\n'
' "1.000000" : 8512,\n'
' "5.000000" : 8640,\n'
' "10.000000" : 8640,\n'
' "20.000000" : 8640,\n'
' "30.000000" : 8768,\n'
' "40.000000" : 8768,\n'
' "50.000000" : 8768,\n'
' "60.000000" : 8768,\n'
' "70.000000" : 8768,\n'
' "80.000000" : 8896,\n'
' "90.000000" : 9024,\n'
' "95.000000" : 9536,\n'
' "99.000000" : 19328,\n'
' "99.500000" : 24960,\n'
' "99.900000" : 26240,\n'
' "99.950000" : 41728,\n'
' "99.990000" : 65280\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9959,\n'
' "max" : 97583,\n'
' "mean" : 11030.278872,\n'
' "stddev" : '
'2148.473123,\n'
' "N" : 620719\n'
' },\n'
' "bw_min" : 353864,\n'
' "bw_max" : 355863,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'354830.928571,\n'
' "bw_dev" : 468.121198,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 88466,\n'
' "iops_max" : 88965,\n'
' "iops_mean" : '
'88707.642857,\n'
' "iops_stddev" : '
'116.917201,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 34.157143,\n'
' "sys_cpu" : 65.814286,\n'
' "ctx" : 19,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 95.638445,\n'
' "20" : 3.521722,\n'
' "50" : 0.793274,\n'
' "100" : 0.039309,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=85581: Tue '
'Oct 25 23:29:49 2022\n'
' read: IOPS=88.7k, BW=346MiB/s '
'(363MB/s)(2425MiB/7001msec)\n'
' slat (nsec): min=1783, '
'max=14928, avg=1926.78, '
'stdev=147.91\n'
' clat (nsec): min=611, '
'max=95610, avg=9103.49, '
'stdev=2144.72\n'
' lat (nsec): min=9959, '
'max=97583, avg=11030.28, '
'stdev=2148.47\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8512], 5.00th=[ '
'8640], 10.00th=[ 8640], 20.00th=[ '
'8640],\n'
' | 30.00th=[ 8768], 40.00th=[ '
'8768], 50.00th=[ 8768], 60.00th=[ '
'8768],\n'
' | 70.00th=[ 8768], 80.00th=[ '
'8896], 90.00th=[ 9024], 95.00th=[ '
'9536],\n'
' | 99.00th=[19328], '
'99.50th=[24960], 99.90th=[26240], '
'99.95th=[41728],\n'
' | 99.99th=[65280]\n'
' bw ( KiB/s): min=353864, '
'max=355863, per=100.00%, '
'avg=354830.93, stdev=468.12, '
'samples=14\n'
' iops : min=88466, '
'max=88965, avg=88707.64, '
'stdev=116.92, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=95.64%, 20=3.52%, 50=0.79%, '
'100=0.04%\n'
' cpu : usr=34.16%, '
'sys=65.81%, ctx=19, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=620719,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=346MiB/s (363MB/s), '
'346MiB/s-346MiB/s '
'(363MB/s-363MB/s), io=2425MiB '
'(2542MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_84.output'),
'state': {'begin': 1666726300.5469594,
'cmd': 'cat '
'/tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7896029949188232,
'end': 1666726301.3365624,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_84.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_84.state')},
'cmd_85': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_85.output'),
'state': {'begin': 1666726303.3699868,
'cmd': 'rm '
'/tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7868921756744385,
'end': 1666726304.156879,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_85.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_85.state')},
'cmd_86': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_86.output'),
'state': {'begin': 1666726304.1589577,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.434263229370117,
'end': 1666726315.593221,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_86.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_86.state')},
'cmd_87': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733404,\n'
' "timestamp_ms" : '
'1666733404980,\n'
' "time" : "Tue Oct 25 23:30:04 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2823823360,\n'
' "io_kbytes" : 2757640,\n'
' "bw_bytes" : 403403337,\n'
' "bw" : 393948,\n'
' "iops" : 98487.142857,\n'
' "runtime" : 7000,\n'
' "total_ios" : 689410,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1582,\n'
' "max" : 36869,\n'
' "mean" : 1652.357284,\n'
' "stddev" : 116.694315,\n'
' "N" : 689410\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 651,\n'
' "max" : 112781,\n'
' "mean" : 8251.688869,\n'
' "stddev" : '
'1885.218026,\n'
' "N" : 689410,\n'
' "percentile" : {\n'
' "1.000000" : 7776,\n'
' "5.000000" : 7776,\n'
' "10.000000" : 7776,\n'
' "20.000000" : 7840,\n'
' "30.000000" : 7840,\n'
' "40.000000" : 7904,\n'
' "50.000000" : 7904,\n'
' "60.000000" : 7904,\n'
' "70.000000" : 8032,\n'
' "80.000000" : 8160,\n'
' "90.000000" : 8256,\n'
' "95.000000" : 8640,\n'
' "99.000000" : 17280,\n'
' "99.500000" : 19328,\n'
' "99.900000" : 22656,\n'
' "99.950000" : 41216,\n'
' "99.990000" : 67072\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9198,\n'
' "max" : 114405,\n'
' "mean" : 9904.046153,\n'
' "stddev" : '
'1889.202305,\n'
' "N" : 689410\n'
' },\n'
' "bw_min" : 393288,\n'
' "bw_max" : 394592,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'393990.571429,\n'
' "bw_dev" : 426.285876,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 98322,\n'
' "iops_max" : 98648,\n'
' "iops_mean" : '
'98497.642857,\n'
' "iops_stddev" : '
'106.571469,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 33.642857,\n'
' "sys_cpu" : 66.328571,\n'
' "ctx" : 17,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.026167,\n'
' "20" : 3.730146,\n'
' "50" : 0.206408,\n'
' "100" : 0.035683,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=85630: Tue '
'Oct 25 23:30:04 2022\n'
' read: IOPS=98.5k, BW=385MiB/s '
'(403MB/s)(2693MiB/7000msec)\n'
' slat (nsec): min=1582, '
'max=36869, avg=1652.36, '
'stdev=116.69\n'
' clat (nsec): min=651, '
'max=112781, avg=8251.69, '
'stdev=1885.22\n'
' lat (usec): min=9, max=114, '
'avg= 9.90, stdev= 1.89\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7776], 5.00th=[ '
'7776], 10.00th=[ 7776], 20.00th=[ '
'7840],\n'
' | 30.00th=[ 7840], 40.00th=[ '
'7904], 50.00th=[ 7904], 60.00th=[ '
'7904],\n'
' | 70.00th=[ 8032], 80.00th=[ '
'8160], 90.00th=[ 8256], 95.00th=[ '
'8640],\n'
' | 99.00th=[17280], '
'99.50th=[19328], 99.90th=[22656], '
'99.95th=[41216],\n'
' | 99.99th=[67072]\n'
' bw ( KiB/s): min=393288, '
'max=394592, per=100.00%, '
'avg=393990.57, stdev=426.29, '
'samples=14\n'
' iops : min=98322, '
'max=98648, avg=98497.64, '
'stdev=106.57, samples=14\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.03%, 20=3.73%, 50=0.21%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=33.64%, '
'sys=66.33%, ctx=17, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=689410,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=385MiB/s (403MB/s), '
'385MiB/s-385MiB/s '
'(403MB/s-403MB/s), io=2693MiB '
'(2824MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_87.output'),
'state': {'begin': 1666726315.5958962,
'cmd': 'cat '
'/tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8652126789093018,
'end': 1666726316.461109,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_87.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_87.state')},
'cmd_88': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_88.output'),
'state': {'begin': 1666726318.6691997,
'cmd': 'rm '
'/tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8573663234710693,
'end': 1666726319.526566,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_88.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_88.state')},
'cmd_89': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_89.output'),
'state': {'begin': 1666726319.5289133,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.511648178100586,
'end': 1666726331.0405614,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_89.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_89.state')},
'cmd_90': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 4096B-4096B, '
'(W) 4096B-4096B, (T) 4096B-4096B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733420,\n'
' "timestamp_ms" : '
'1666733420440,\n'
' "time" : "Tue Oct 25 23:30:20 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 2837221376,\n'
' "io_kbytes" : 2770724,\n'
' "bw_bytes" : 405259445,\n'
' "bw" : 395761,\n'
' "iops" : 98940.294244,\n'
' "runtime" : 7001,\n'
' "total_ios" : 692681,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1563,\n'
' "max" : 52007,\n'
' "mean" : 1627.296386,\n'
' "stddev" : 124.623637,\n'
' "N" : 692681\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 631,\n'
' "max" : 93886,\n'
' "mean" : 8236.372760,\n'
' "stddev" : '
'1870.436580,\n'
' "N" : 692681,\n'
' "percentile" : {\n'
' "1.000000" : 7584,\n'
' "5.000000" : 7648,\n'
' "10.000000" : 7840,\n'
' "20.000000" : 7904,\n'
' "30.000000" : 7904,\n'
' "40.000000" : 7904,\n'
' "50.000000" : 7968,\n'
' "60.000000" : 7968,\n'
' "70.000000" : 7968,\n'
' "80.000000" : 7968,\n'
' "90.000000" : 8032,\n'
' "95.000000" : 8640,\n'
' "99.000000" : 17280,\n'
' "99.500000" : 19328,\n'
' "99.900000" : 22656,\n'
' "99.950000" : 40704,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9167,\n'
' "max" : 95499,\n'
' "mean" : 9863.669146,\n'
' "stddev" : '
'1874.856856,\n'
' "N" : 692681\n'
' },\n'
' "bw_min" : 395336,\n'
' "bw_max" : 396672,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'395852.500000,\n'
' "bw_dev" : 386.086230,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 98834,\n'
' "iops_max" : 99168,\n'
' "iops_mean" : '
'98963.071429,\n'
' "iops_stddev" : '
'96.554858,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 28.842857,\n'
' "sys_cpu" : 71.128571,\n'
' "ctx" : 44,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.036992,\n'
' "20" : 3.718595,\n'
' "50" : 0.209476,\n'
' "100" : 0.033926,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=85679: Tue '
'Oct 25 23:30:20 2022\n'
' read: IOPS=98.9k, BW=386MiB/s '
'(405MB/s)(2706MiB/7001msec)\n'
' slat (nsec): min=1563, '
'max=52007, avg=1627.30, '
'stdev=124.62\n'
' clat (nsec): min=631, '
'max=93886, avg=8236.37, '
'stdev=1870.44\n'
' lat (nsec): min=9167, '
'max=95499, avg=9863.67, '
'stdev=1874.86\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7584], 5.00th=[ '
'7648], 10.00th=[ 7840], 20.00th=[ '
'7904],\n'
' | 30.00th=[ 7904], 40.00th=[ '
'7904], 50.00th=[ 7968], 60.00th=[ '
'7968],\n'
' | 70.00th=[ 7968], 80.00th=[ '
'7968], 90.00th=[ 8032], 95.00th=[ '
'8640],\n'
' | 99.00th=[17280], '
'99.50th=[19328], 99.90th=[22656], '
'99.95th=[40704],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=395336, '
'max=396672, per=100.00%, '
'avg=395852.50, stdev=386.09, '
'samples=14\n'
' iops : min=98834, '
'max=99168, avg=98963.07, '
'stdev=96.55, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 10=96.04%, '
'20=3.72%, 50=0.21%, 100=0.03%\n'
' cpu : usr=28.84%, '
'sys=71.13%, ctx=44, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=692681,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=386MiB/s (405MB/s), '
'386MiB/s-386MiB/s '
'(405MB/s-405MB/s), io=2706MiB '
'(2837MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_90.output'),
'state': {'begin': 1666726331.043958,
'cmd': 'cat '
'/tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7904596328735352,
'end': 1666726331.8344176,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_90.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_90.state')},
'cmd_91': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_libaio-xnvme_0.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_91.output'),
'state': {'begin': 1666726332.8733482,
'cmd': 'rm '
'/tmp/fio-output_4K_1_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7861638069152832,
'end': 1666726333.659512,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_91.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_91.state')},
'cmd_92': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_92.output'),
'state': {'begin': 1666726333.6615798,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_libaio-xnvme_0.txt"',
'cwd': 'None',
'elapsed': 11.092099666595459,
'end': 1666726344.7536795,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_92.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_92.state')},
'cmd_93': {'output': 'libaio-xnvme: (g=0): rw=randread, '
'bs=(R) 4096B-4096B, (W) '
'4096B-4096B, (T) 4096B-4096B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733434,\n'
' "timestamp_ms" : '
'1666733434249,\n'
' "time" : "Tue Oct 25 23:30:34 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : "libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1678741504,\n'
' "io_kbytes" : 1639396,\n'
' "bw_bytes" : 239785959,\n'
' "bw" : 234165,\n'
' "iops" : 58541.494072,\n'
' "runtime" : 7001,\n'
' "total_ios" : 409849,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2966,\n'
' "max" : 24035,\n'
' "mean" : 3340.552930,\n'
' "stddev" : 206.631663,\n'
' "N" : 409849\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1123,\n'
' "max" : 438051,\n'
' "mean" : 13173.901883,\n'
' "stddev" : '
'2319.222617,\n'
' "N" : 409849,\n'
' "percentile" : {\n'
' "1.000000" : 11968,\n'
' "5.000000" : 12096,\n'
' "10.000000" : 12096,\n'
' "20.000000" : 12224,\n'
' "30.000000" : 12352,\n'
' "40.000000" : 12480,\n'
' "50.000000" : 13376,\n'
' "60.000000" : 13376,\n'
' "70.000000" : 13376,\n'
' "80.000000" : 13504,\n'
' "90.000000" : 13504,\n'
' "95.000000" : 13632,\n'
' "99.000000" : 23680,\n'
' "99.500000" : 28800,\n'
' "99.900000" : 35072,\n'
' "99.950000" : 50944,\n'
' "99.990000" : 73216\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12203,\n'
' "max" : 441297,\n'
' "mean" : 16514.454814,\n'
' "stddev" : '
'2343.548276,\n'
' "N" : 409849\n'
' },\n'
' "bw_min" : 226544,\n'
' "bw_max" : 245707,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'234225.357143,\n'
' "bw_dev" : 8659.960984,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 56636,\n'
' "iops_max" : 61426,\n'
' "iops_mean" : '
'58556.285714,\n'
' "iops_stddev" : '
'2164.913764,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 12.657143,\n'
' "sys_cpu" : 36.671429,\n'
' "ctx" : 409826,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 98.453089,\n'
' "50" : 1.489085,\n'
' "100" : 0.050506,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=85718: Tue '
'Oct 25 23:30:34 2022\n'
' read: IOPS=58.5k, BW=229MiB/s '
'(240MB/s)(1601MiB/7001msec)\n'
' slat (nsec): min=2966, '
'max=24035, avg=3340.55, '
'stdev=206.63\n'
' clat (nsec): min=1123, '
'max=438051, avg=13173.90, '
'stdev=2319.22\n'
' lat (usec): min=12, max=441, '
'avg=16.51, stdev= 2.34\n'
' clat percentiles (nsec):\n'
' | 1.00th=[11968], '
'5.00th=[12096], 10.00th=[12096], '
'20.00th=[12224],\n'
' | 30.00th=[12352], '
'40.00th=[12480], 50.00th=[13376], '
'60.00th=[13376],\n'
' | 70.00th=[13376], '
'80.00th=[13504], 90.00th=[13504], '
'95.00th=[13632],\n'
' | 99.00th=[23680], '
'99.50th=[28800], 99.90th=[35072], '
'99.95th=[50944],\n'
' | 99.99th=[73216]\n'
' bw ( KiB/s): min=226544, '
'max=245707, per=100.00%, '
'avg=234225.36, stdev=8659.96, '
'samples=14\n'
' iops : min=56636, '
'max=61426, avg=58556.29, '
'stdev=2164.91, samples=14\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.01%, 20=98.45%, '
'50=1.49%\n'
' lat (usec) : 100=0.05%, '
'250=0.01%, 500=0.01%\n'
' cpu : usr=12.66%, '
'sys=36.67%, ctx=409826, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=409849,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=229MiB/s (240MB/s), '
'229MiB/s-229MiB/s '
'(240MB/s-240MB/s), io=1601MiB '
'(1679MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_93.output'),
'state': {'begin': 1666726344.7567978,
'cmd': 'cat '
'/tmp/fio-output_4K_1_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8069570064544678,
'end': 1666726345.5637548,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_93.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_93.state')},
'cmd_94': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_libaio-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_94.output'),
'state': {'begin': 1666726346.6274297,
'cmd': 'rm '
'/tmp/fio-output_4K_1_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7909464836120605,
'end': 1666726347.4183762,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_94.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_94.state')},
'cmd_95': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_95.output'),
'state': {'begin': 1666726347.420631,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_libaio-xnvme_1.txt"',
'cwd': 'None',
'elapsed': 11.116731643676758,
'end': 1666726358.5373626,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_95.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_95.state')},
'cmd_96': {'output': 'libaio-xnvme: (g=0): rw=randread, '
'bs=(R) 4096B-4096B, (W) '
'4096B-4096B, (T) 4096B-4096B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733448,\n'
' "timestamp_ms" : '
'1666733448033,\n'
' "time" : "Tue Oct 25 23:30:48 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : "libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1764671488,\n'
' "io_kbytes" : 1723312,\n'
' "bw_bytes" : 252059918,\n'
' "bw" : 246152,\n'
' "iops" : 61538.065991,\n'
' "runtime" : 7001,\n'
' "total_ios" : 430828,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2925,\n'
' "max" : 175278,\n'
' "mean" : 3159.793955,\n'
' "stddev" : 387.428894,\n'
' "N" : 430828\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1002,\n'
' "max" : 182192,\n'
' "mean" : 12504.355956,\n'
' "stddev" : '
'2223.696600,\n'
' "N" : 430828,\n'
' "percentile" : {\n'
' "1.000000" : 11968,\n'
' "5.000000" : 11968,\n'
' "10.000000" : 12096,\n'
' "20.000000" : 12096,\n'
' "30.000000" : 12096,\n'
' "40.000000" : 12224,\n'
' "50.000000" : 12224,\n'
' "60.000000" : 12224,\n'
' "70.000000" : 12352,\n'
' "80.000000" : 12352,\n'
' "90.000000" : 12480,\n'
' "95.000000" : 12608,\n'
' "99.000000" : 22144,\n'
' "99.500000" : 29056,\n'
' "99.900000" : 37632,\n'
' "99.950000" : 52992,\n'
' "99.990000" : 73216\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12213,\n'
' "max" : 188343,\n'
' "mean" : 15664.149911,\n'
' "stddev" : '
'2259.446273,\n'
' "N" : 430828\n'
' },\n'
' "bw_min" : 245008,\n'
' "bw_max" : 247408,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'246227.357143,\n'
' "bw_dev" : 707.156343,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 61252,\n'
' "iops_max" : 61852,\n'
' "iops_mean" : '
'61556.785714,\n'
' "iops_stddev" : '
'176.840727,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.242857,\n'
' "sys_cpu" : 39.485714,\n'
' "ctx" : 430779,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010213,\n'
' "20" : 98.469227,\n'
' "50" : 1.455801,\n'
' "100" : 0.060581,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=85758: Tue '
'Oct 25 23:30:48 2022\n'
' read: IOPS=61.5k, BW=240MiB/s '
'(252MB/s)(1683MiB/7001msec)\n'
' slat (nsec): min=2925, '
'max=175278, avg=3159.79, '
'stdev=387.43\n'
' clat (nsec): min=1002, '
'max=182192, avg=12504.36, '
'stdev=2223.70\n'
' lat (usec): min=12, max=188, '
'avg=15.66, stdev= 2.26\n'
' clat percentiles (nsec):\n'
' | 1.00th=[11968], '
'5.00th=[11968], 10.00th=[12096], '
'20.00th=[12096],\n'
' | 30.00th=[12096], '
'40.00th=[12224], 50.00th=[12224], '
'60.00th=[12224],\n'
' | 70.00th=[12352], '
'80.00th=[12352], 90.00th=[12480], '
'95.00th=[12608],\n'
' | 99.00th=[22144], '
'99.50th=[29056], 99.90th=[37632], '
'99.95th=[52992],\n'
' | 99.99th=[73216]\n'
' bw ( KiB/s): min=245008, '
'max=247408, per=100.00%, '
'avg=246227.36, stdev=707.16, '
'samples=14\n'
' iops : min=61252, '
'max=61852, avg=61556.79, '
'stdev=176.84, samples=14\n'
' lat (usec) : 2=0.01%, '
'10=0.01%, 20=98.47%, 50=1.46%, '
'100=0.06%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=13.24%, '
'sys=39.49%, ctx=430779, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=430828,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=240MiB/s (252MB/s), '
'240MiB/s-240MiB/s '
'(252MB/s-252MB/s), io=1683MiB '
'(1765MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_96.output'),
'state': {'begin': 1666726358.5440695,
'cmd': 'cat '
'/tmp/fio-output_4K_1_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7939703464508057,
'end': 1666726359.3380399,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_96.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_96.state')},
'cmd_97': {'output': 'rm: cannot remove '
"'/tmp/fio-output_4K_1_libaio-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_97.output'),
'state': {'begin': 1666726360.324495,
'cmd': 'rm '
'/tmp/fio-output_4K_1_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7884197235107422,
'end': 1666726361.1129148,
'err': 1,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_97.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_97.state')},
'cmd_98': {'output': '',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_98.output'),
'state': {'begin': 1666726361.1157036,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="4K" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_4K_1_libaio-xnvme_2.txt"',
'cwd': 'None',
'elapsed': 11.09414291381836,
'end': 1666726372.2098465,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_98.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_98.state')},
'cmd_99': {'output': 'libaio-xnvme: (g=0): rw=randread, '
'bs=(R) 4096B-4096B, (W) '
'4096B-4096B, (T) 4096B-4096B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666733461,\n'
' "timestamp_ms" : '
'1666733461706,\n'
' "time" : "Tue Oct 25 23:31:01 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : "libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "4K",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 1754910720,\n'
' "io_kbytes" : 1713780,\n'
' "bw_bytes" : 250665722,\n'
' "bw" : 244790,\n'
' "iops" : 61197.686045,\n'
' "runtime" : 7001,\n'
' "total_ios" : 428445,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2905,\n'
' "max" : 25257,\n'
' "mean" : 3138.958107,\n'
' "stddev" : 264.649411,\n'
' "N" : 428445\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 1072,\n'
' "max" : 196789,\n'
' "mean" : 12600.985377,\n'
' "stddev" : '
'2183.473992,\n'
' "N" : 428445,\n'
' "percentile" : {\n'
' "1.000000" : 11968,\n'
' "5.000000" : 12096,\n'
' "10.000000" : 12096,\n'
' "20.000000" : 12224,\n'
' "30.000000" : 12224,\n'
' "40.000000" : 12224,\n'
' "50.000000" : 12352,\n'
' "60.000000" : 12352,\n'
' "70.000000" : 12480,\n'
' "80.000000" : 12480,\n'
' "90.000000" : 12480,\n'
' "95.000000" : 12736,\n'
' "99.000000" : 22400,\n'
' "99.500000" : 29312,\n'
' "99.900000" : 32384,\n'
' "99.950000" : 50944,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12002,\n'
' "max" : 199895,\n'
' "mean" : 15739.943484,\n'
' "stddev" : '
'2203.074385,\n'
' "N" : 428445\n'
' },\n'
' "bw_min" : 243672,\n'
' "bw_max" : 246248,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'244847.571429,\n'
' "bw_dev" : 682.736655,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 60918,\n'
' "iops_max" : 61562,\n'
' "iops_mean" : '
'61211.857143,\n'
' "iops_stddev" : '
'170.666502,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 12.700000,\n'
' "sys_cpu" : 39.757143,\n'
' "ctx" : 428394,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 98.445308,\n'
' "50" : 1.491907,\n'
' "100" : 0.050415,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=85798: Tue '
'Oct 25 23:31:01 2022\n'
' read: IOPS=61.2k, BW=239MiB/s '
'(251MB/s)(1674MiB/7001msec)\n'
' slat (nsec): min=2905, '
'max=25257, avg=3138.96, '
'stdev=264.65\n'
' clat (nsec): min=1072, '
'max=196789, avg=12600.99, '
'stdev=2183.47\n'
' lat (usec): min=12, max=199, '
'avg=15.74, stdev= 2.20\n'
' clat percentiles (nsec):\n'
' | 1.00th=[11968], '
'5.00th=[12096], 10.00th=[12096], '
'20.00th=[12224],\n'
' | 30.00th=[12224], '
'40.00th=[12224], 50.00th=[12352], '
'60.00th=[12352],\n'
' | 70.00th=[12480], '
'80.00th=[12480], 90.00th=[12480], '
'95.00th=[12736],\n'
' | 99.00th=[22400], '
'99.50th=[29312], 99.90th=[32384], '
'99.95th=[50944],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=243672, '
'max=246248, per=100.00%, '
'avg=244847.57, stdev=682.74, '
'samples=14\n'
' iops : min=60918, '
'max=61562, avg=61211.86, '
'stdev=170.67, samples=14\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.01%, 20=98.45%, '
'50=1.49%\n'
' lat (usec) : 100=0.05%, '
'250=0.01%\n'
' cpu : usr=12.70%, '
'sys=39.76%, ctx=428394, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=428445,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=239MiB/s (251MB/s), '
'239MiB/s-239MiB/s '
'(251MB/s-251MB/s), io=1674MiB '
'(1755MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_99.output'),
'state': {'begin': 1666726372.2128544,
'cmd': 'cat '
'/tmp/fio-output_4K_1_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.78910231590271,
'end': 1666726373.0019567,
'err': 0,
'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_99.output'},
'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_99.state')}}},
'id': '002_fio',
'name': 'fio',
'nr': 2,
'status': {'elapsed': 1890.7196505069733,
'failed': 0,
'passed': 1,
'skipped': 0,
'started': 0.0},
'uses': 'fioe',
'with': {'iodepths': [1, 2, 4, 8], 'iosizes': ['4K'], 'repetitions': 3}}
$ rm /tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt"
$ cat /tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733002,
"timestamp_ms" : 1666733002638,
"time" : "Tue Oct 25 23:23:22 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 1880924160,
"io_kbytes" : 1836840,
"bw_bytes" : 268665070,
"bw" : 262368,
"iops" : 65591.915441,
"runtime" : 7001,
"total_ios" : 459209,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2184,
"max" : 28192,
"mean" : 2371.134109,
"stddev" : 209.422890,
"N" : 459210
},
"clat_ns" : {
"min" : 241,
"max" : 1211963,
"mean" : 12255.930946,
"stddev" : 2652.769239,
"N" : 459209,
"percentile" : {
"1.000000" : 11712,
"5.000000" : 11712,
"10.000000" : 11840,
"20.000000" : 11840,
"30.000000" : 11840,
"40.000000" : 11968,
"50.000000" : 11968,
"60.000000" : 12096,
"70.000000" : 12096,
"80.000000" : 12096,
"90.000000" : 12224,
"95.000000" : 12352,
"99.000000" : 21888,
"99.500000" : 23424,
"99.900000" : 31872,
"99.950000" : 50432,
"99.990000" : 64256
}
},
"lat_ns" : {
"min" : 11632,
"max" : 1214328,
"mean" : 14627.064896,
"stddev" : 2660.860079,
"N" : 459209
},
"bw_min" : 260360,
"bw_max" : 263680,
"bw_agg" : 100.000000,
"bw_mean" : 262437.714286,
"bw_dev" : 778.330702,
"bw_samples" : 14,
"iops_min" : 65090,
"iops_max" : 65920,
"iops_mean" : 65609.428571,
"iops_stddev" : 194.582675,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 9.857143,
"sys_cpu" : 36.157143,
"ctx" : 459183,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 98.399857,
"50" : 1.534595,
"100" : 0.054877,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=84392: Tue Oct 25 23:23:22 2022
read: IOPS=65.6k, BW=256MiB/s (269MB/s)(1794MiB/7001msec)
slat (nsec): min=2184, max=28192, avg=2371.13, stdev=209.42
clat (nsec): min=241, max=1212.0k, avg=12255.93, stdev=2652.77
lat (usec): min=11, max=1214, avg=14.63, stdev= 2.66
clat percentiles (nsec):
| 1.00th=[11712], 5.00th=[11712], 10.00th=[11840], 20.00th=[11840],
| 30.00th=[11840], 40.00th=[11968], 50.00th=[11968], 60.00th=[12096],
| 70.00th=[12096], 80.00th=[12096], 90.00th=[12224], 95.00th=[12352],
| 99.00th=[21888], 99.50th=[23424], 99.90th=[31872], 99.95th=[50432],
| 99.99th=[64256]
bw ( KiB/s): min=260360, max=263680, per=100.00%, avg=262437.71, stdev=778.33, samples=14
iops : min=65090, max=65920, avg=65609.43, stdev=194.58, samples=14
lat (nsec) : 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=98.40%, 50=1.53%, 100=0.05%
lat (usec) : 250=0.01%
lat (msec) : 2=0.01%
cpu : usr=9.86%, sys=36.16%, ctx=459183, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=459209,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=256MiB/s (269MB/s), 256MiB/s-256MiB/s (269MB/s-269MB/s), io=1794MiB (1881MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt"
$ cat /tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733016,
"timestamp_ms" : 1666733016245,
"time" : "Tue Oct 25 23:23:36 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 1887145984,
"io_kbytes" : 1842916,
"bw_bytes" : 269553775,
"bw" : 263236,
"iops" : 65809.027282,
"runtime" : 7001,
"total_ios" : 460729,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2174,
"max" : 31970,
"mean" : 2353.754324,
"stddev" : 207.012099,
"N" : 460729
},
"clat_ns" : {
"min" : 200,
"max" : 183274,
"mean" : 12205.161577,
"stddev" : 1978.440122,
"N" : 460729,
"percentile" : {
"1.000000" : 11712,
"5.000000" : 11712,
"10.000000" : 11712,
"20.000000" : 11840,
"30.000000" : 11840,
"40.000000" : 11840,
"50.000000" : 11968,
"60.000000" : 11968,
"70.000000" : 12096,
"80.000000" : 12096,
"90.000000" : 12096,
"95.000000" : 12352,
"99.000000" : 21888,
"99.500000" : 23168,
"99.900000" : 31360,
"99.950000" : 49920,
"99.990000" : 64256
}
},
"lat_ns" : {
"min" : 11591,
"max" : 185708,
"mean" : 14558.915901,
"stddev" : 1988.143930,
"N" : 460729
},
"bw_min" : 262208,
"bw_max" : 263920,
"bw_agg" : 100.000000,
"bw_mean" : 263299.928571,
"bw_dev" : 429.320118,
"bw_samples" : 14,
"iops_min" : 65552,
"iops_max" : 65980,
"iops_mean" : 65824.928571,
"iops_stddev" : 107.268509,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 9.728571,
"sys_cpu" : 36.385714,
"ctx" : 460697,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 98.392330,
"50" : 1.542989,
"100" : 0.051874,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=84430: Tue Oct 25 23:23:36 2022
read: IOPS=65.8k, BW=257MiB/s (270MB/s)(1800MiB/7001msec)
slat (nsec): min=2174, max=31970, avg=2353.75, stdev=207.01
clat (nsec): min=200, max=183274, avg=12205.16, stdev=1978.44
lat (usec): min=11, max=185, avg=14.56, stdev= 1.99
clat percentiles (nsec):
| 1.00th=[11712], 5.00th=[11712], 10.00th=[11712], 20.00th=[11840],
| 30.00th=[11840], 40.00th=[11840], 50.00th=[11968], 60.00th=[11968],
| 70.00th=[12096], 80.00th=[12096], 90.00th=[12096], 95.00th=[12352],
| 99.00th=[21888], 99.50th=[23168], 99.90th=[31360], 99.95th=[49920],
| 99.99th=[64256]
bw ( KiB/s): min=262208, max=263920, per=100.00%, avg=263299.93, stdev=429.32, samples=14
iops : min=65552, max=65980, avg=65824.93, stdev=107.27, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 10=0.01%, 20=98.39%, 50=1.54%, 100=0.05%, 250=0.01%
cpu : usr=9.73%, sys=36.39%, ctx=460697, majf=0, minf=1
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=460729,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=257MiB/s (270MB/s), 257MiB/s-257MiB/s (270MB/s-270MB/s), io=1800MiB (1887MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt"
$ cat /tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733029,
"timestamp_ms" : 1666733029933,
"time" : "Tue Oct 25 23:23:49 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 1777991680,
"io_kbytes" : 1736320,
"bw_bytes" : 253962531,
"bw" : 248010,
"iops" : 62002.571061,
"runtime" : 7001,
"total_ios" : 434080,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2184,
"max" : 31269,
"mean" : 2476.576864,
"stddev" : 207.606246,
"N" : 434080
},
"clat_ns" : {
"min" : 250,
"max" : 133119,
"mean" : 13071.487523,
"stddev" : 2165.425154,
"N" : 434080,
"percentile" : {
"1.000000" : 11968,
"5.000000" : 11968,
"10.000000" : 12096,
"20.000000" : 12096,
"30.000000" : 12096,
"40.000000" : 12224,
"50.000000" : 13248,
"60.000000" : 13376,
"70.000000" : 13376,
"80.000000" : 13376,
"90.000000" : 13504,
"95.000000" : 13632,
"99.000000" : 22912,
"99.500000" : 29312,
"99.900000" : 32640,
"99.950000" : 50432,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 13816,
"max" : 135424,
"mean" : 15548.064387,
"stddev" : 2206.175957,
"N" : 434080
},
"bw_min" : 237664,
"bw_max" : 261000,
"bw_agg" : 100.000000,
"bw_mean" : 248095.357143,
"bw_dev" : 11422.448246,
"bw_samples" : 14,
"iops_min" : 59416,
"iops_max" : 65250,
"iops_mean" : 62023.785714,
"iops_stddev" : 2855.663204,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 8.014286,
"sys_cpu" : 35.057143,
"ctx" : 434068,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 98.284187,
"50" : 1.652691,
"100" : 0.052525,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=84468: Tue Oct 25 23:23:49 2022
read: IOPS=62.0k, BW=242MiB/s (254MB/s)(1696MiB/7001msec)
slat (nsec): min=2184, max=31269, avg=2476.58, stdev=207.61
clat (nsec): min=250, max=133119, avg=13071.49, stdev=2165.43
lat (usec): min=13, max=135, avg=15.55, stdev= 2.21
clat percentiles (nsec):
| 1.00th=[11968], 5.00th=[11968], 10.00th=[12096], 20.00th=[12096],
| 30.00th=[12096], 40.00th=[12224], 50.00th=[13248], 60.00th=[13376],
| 70.00th=[13376], 80.00th=[13376], 90.00th=[13504], 95.00th=[13632],
| 99.00th=[22912], 99.50th=[29312], 99.90th=[32640], 99.95th=[50432],
| 99.99th=[68096]
bw ( KiB/s): min=237664, max=261000, per=100.00%, avg=248095.36, stdev=11422.45, samples=14
iops : min=59416, max=65250, avg=62023.79, stdev=2855.66, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 10=0.01%, 20=98.28%, 50=1.65%, 100=0.05%, 250=0.01%
cpu : usr=8.01%, sys=35.06%, ctx=434068, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=434080,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=242MiB/s (254MB/s), 242MiB/s-242MiB/s (254MB/s-254MB/s), io=1696MiB (1778MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt': No such file or directory
$ rm /tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt"
$ cat /tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733475,
"timestamp_ms" : 1666733475316,
"time" : "Tue Oct 25 23:31:15 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 2836426752,
"io_kbytes" : 2769948,
"bw_bytes" : 405145943,
"bw" : 395650,
"iops" : 98912.298243,
"runtime" : 7001,
"total_ios" : 692485,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1893,
"max" : 27962,
"mean" : 2200.003763,
"stddev" : 323.295581,
"N" : 692486
},
"clat_ns" : {
"min" : 3106,
"max" : 189856,
"mean" : 17485.482183,
"stddev" : 3150.118639,
"N" : 692486,
"percentile" : {
"1.000000" : 16320,
"5.000000" : 16512,
"10.000000" : 16512,
"20.000000" : 16512,
"30.000000" : 16512,
"40.000000" : 16768,
"50.000000" : 17024,
"60.000000" : 17024,
"70.000000" : 17024,
"80.000000" : 17280,
"90.000000" : 17280,
"95.000000" : 19584,
"99.000000" : 33536,
"99.500000" : 34560,
"99.900000" : 50432,
"99.950000" : 62720,
"99.990000" : 73216
}
},
"lat_ns" : {
"min" : 11261,
"max" : 191810,
"mean" : 19685.485946,
"stddev" : 3142.083143,
"N" : 692486
},
"bw_min" : 394512,
"bw_max" : 396576,
"bw_agg" : 100.000000,
"bw_mean" : 395746.285714,
"bw_dev" : 618.185363,
"bw_samples" : 14,
"iops_min" : 98628,
"iops_max" : 99144,
"iops_mean" : 98936.571429,
"iops_stddev" : 154.546341,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 12.085714,
"sys_cpu" : 38.200000,
"ctx" : 346121,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010686,
"20" : 95.026318,
"50" : 4.855268,
"100" : 0.104840,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=85837: Tue Oct 25 23:31:15 2022
read: IOPS=98.9k, BW=386MiB/s (405MB/s)(2705MiB/7001msec)
slat (nsec): min=1893, max=27962, avg=2200.00, stdev=323.30
clat (usec): min=3, max=189, avg=17.49, stdev= 3.15
lat (usec): min=11, max=191, avg=19.69, stdev= 3.14
clat percentiles (nsec):
| 1.00th=[16320], 5.00th=[16512], 10.00th=[16512], 20.00th=[16512],
| 30.00th=[16512], 40.00th=[16768], 50.00th=[17024], 60.00th=[17024],
| 70.00th=[17024], 80.00th=[17280], 90.00th=[17280], 95.00th=[19584],
| 99.00th=[33536], 99.50th=[34560], 99.90th=[50432], 99.95th=[62720],
| 99.99th=[73216]
bw ( KiB/s): min=394512, max=396576, per=100.00%, avg=395746.29, stdev=618.19, samples=14
iops : min=98628, max=99144, avg=98936.57, stdev=154.55, samples=14
lat (usec) : 4=0.01%, 10=0.01%, 20=95.03%, 50=4.86%, 100=0.10%
lat (usec) : 250=0.01%
cpu : usr=12.09%, sys=38.20%, ctx=346121, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=692485,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=386MiB/s (405MB/s), 386MiB/s-386MiB/s (405MB/s-405MB/s), io=2705MiB (2836MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt"
$ cat /tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733488,
"timestamp_ms" : 1666733488966,
"time" : "Tue Oct 25 23:31:28 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 2790170624,
"io_kbytes" : 2724776,
"bw_bytes" : 398538869,
"bw" : 389198,
"iops" : 97299.242965,
"runtime" : 7001,
"total_ios" : 681192,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1963,
"max" : 37029,
"mean" : 2279.325263,
"stddev" : 339.765872,
"N" : 681193
},
"clat_ns" : {
"min" : 3075,
"max" : 121388,
"mean" : 17720.137531,
"stddev" : 3211.933311,
"N" : 681193,
"percentile" : {
"1.000000" : 16512,
"5.000000" : 16512,
"10.000000" : 16768,
"20.000000" : 16768,
"30.000000" : 16768,
"40.000000" : 16768,
"50.000000" : 17280,
"60.000000" : 17280,
"70.000000" : 17280,
"80.000000" : 17536,
"90.000000" : 17536,
"95.000000" : 20096,
"99.000000" : 34048,
"99.500000" : 35072,
"99.900000" : 50432,
"99.950000" : 63232,
"99.990000" : 74240
}
},
"lat_ns" : {
"min" : 11381,
"max" : 123462,
"mean" : 19999.462794,
"stddev" : 3203.796715,
"N" : 681193
},
"bw_min" : 388424,
"bw_max" : 390448,
"bw_agg" : 100.000000,
"bw_mean" : 389292.714286,
"bw_dev" : 718.143593,
"bw_samples" : 14,
"iops_min" : 97106,
"iops_max" : 97612,
"iops_mean" : 97323.142857,
"iops_stddev" : 179.542520,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 11.500000,
"sys_cpu" : 39.285714,
"ctx" : 340448,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.011304,
"20" : 94.985261,
"50" : 4.893921,
"100" : 0.104082,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=85875: Tue Oct 25 23:31:28 2022
read: IOPS=97.3k, BW=380MiB/s (399MB/s)(2661MiB/7001msec)
slat (nsec): min=1963, max=37029, avg=2279.33, stdev=339.77
clat (usec): min=3, max=121, avg=17.72, stdev= 3.21
lat (usec): min=11, max=123, avg=20.00, stdev= 3.20
clat percentiles (nsec):
| 1.00th=[16512], 5.00th=[16512], 10.00th=[16768], 20.00th=[16768],
| 30.00th=[16768], 40.00th=[16768], 50.00th=[17280], 60.00th=[17280],
| 70.00th=[17280], 80.00th=[17536], 90.00th=[17536], 95.00th=[20096],
| 99.00th=[34048], 99.50th=[35072], 99.90th=[50432], 99.95th=[63232],
| 99.99th=[74240]
bw ( KiB/s): min=388424, max=390448, per=100.00%, avg=389292.71, stdev=718.14, samples=14
iops : min=97106, max=97612, avg=97323.14, stdev=179.54, samples=14
lat (usec) : 4=0.01%, 10=0.01%, 20=94.99%, 50=4.89%, 100=0.10%
lat (usec) : 250=0.01%
cpu : usr=11.50%, sys=39.29%, ctx=340448, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=681192,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=380MiB/s (399MB/s), 380MiB/s-380MiB/s (399MB/s-399MB/s), io=2661MiB (2790MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt"
$ cat /tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733502,
"timestamp_ms" : 1666733502622,
"time" : "Tue Oct 25 23:31:42 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 2812887040,
"io_kbytes" : 2746960,
"bw_bytes" : 401783608,
"bw" : 392366,
"iops" : 98091.558349,
"runtime" : 7001,
"total_ios" : 686739,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1884,
"max" : 31248,
"mean" : 2222.947482,
"stddev" : 326.104416,
"N" : 686739
},
"clat_ns" : {
"min" : 2996,
"max" : 255970,
"mean" : 17635.028845,
"stddev" : 3256.648995,
"N" : 686740,
"percentile" : {
"1.000000" : 16512,
"5.000000" : 16512,
"10.000000" : 16512,
"20.000000" : 16768,
"30.000000" : 16768,
"40.000000" : 16768,
"50.000000" : 17024,
"60.000000" : 17280,
"70.000000" : 17280,
"80.000000" : 17280,
"90.000000" : 17536,
"95.000000" : 20096,
"99.000000" : 34048,
"99.500000" : 34560,
"99.900000" : 50944,
"99.950000" : 63744,
"99.990000" : 74240
}
},
"lat_ns" : {
"min" : 11231,
"max" : 257904,
"mean" : 19857.976051,
"stddev" : 3249.803901,
"N" : 686740
},
"bw_min" : 388705,
"bw_max" : 395896,
"bw_agg" : 100.000000,
"bw_mean" : 392460.642857,
"bw_dev" : 2587.148404,
"bw_samples" : 14,
"iops_min" : 97176,
"iops_max" : 98974,
"iops_mean" : 98115.142857,
"iops_stddev" : 646.815020,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 12.271429,
"sys_cpu" : 38.271429,
"ctx" : 343225,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.014416,
"20" : 94.981354,
"50" : 4.886718,
"100" : 0.113435,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=85913: Tue Oct 25 23:31:42 2022
read: IOPS=98.1k, BW=383MiB/s (402MB/s)(2683MiB/7001msec)
slat (nsec): min=1884, max=31248, avg=2222.95, stdev=326.10
clat (usec): min=2, max=255, avg=17.64, stdev= 3.26
lat (usec): min=11, max=257, avg=19.86, stdev= 3.25
clat percentiles (nsec):
| 1.00th=[16512], 5.00th=[16512], 10.00th=[16512], 20.00th=[16768],
| 30.00th=[16768], 40.00th=[16768], 50.00th=[17024], 60.00th=[17280],
| 70.00th=[17280], 80.00th=[17280], 90.00th=[17536], 95.00th=[20096],
| 99.00th=[34048], 99.50th=[34560], 99.90th=[50944], 99.95th=[63744],
| 99.99th=[74240]
bw ( KiB/s): min=388705, max=395896, per=100.00%, avg=392460.64, stdev=2587.15, samples=14
iops : min=97176, max=98974, avg=98115.14, stdev=646.82, samples=14
lat (usec) : 4=0.01%, 10=0.01%, 20=94.98%, 50=4.89%, 100=0.11%
lat (usec) : 250=0.01%, 500=0.01%
cpu : usr=12.27%, sys=38.27%, ctx=343225, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=686739,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=383MiB/s (402MB/s), 383MiB/s-383MiB/s (402MB/s-402MB/s), io=2683MiB (2813MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt"
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt"
$ cat /tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733516,
"timestamp_ms" : 1666733516277,
"time" : "Tue Oct 25 23:31:56 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 2816339968,
"io_kbytes" : 2750332,
"bw_bytes" : 402276813,
"bw" : 392848,
"iops" : 98211.969719,
"runtime" : 7001,
"total_ios" : 687582,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2003,
"max" : 40265,
"mean" : 2351.189732,
"stddev" : 337.786924,
"N" : 687582
},
"clat_ns" : {
"min" : 3356,
"max" : 209222,
"mean" : 17502.922175,
"stddev" : 3232.779953,
"N" : 687583,
"percentile" : {
"1.000000" : 16320,
"5.000000" : 16512,
"10.000000" : 16512,
"20.000000" : 16512,
"30.000000" : 16512,
"40.000000" : 16768,
"50.000000" : 17024,
"60.000000" : 17024,
"70.000000" : 17280,
"80.000000" : 17280,
"90.000000" : 17280,
"95.000000" : 19584,
"99.000000" : 33536,
"99.500000" : 34560,
"99.900000" : 50432,
"99.950000" : 63744,
"99.990000" : 74240
}
},
"lat_ns" : {
"min" : 11251,
"max" : 211306,
"mean" : 19854.111460,
"stddev" : 3223.695868,
"N" : 687583
},
"bw_min" : 391559,
"bw_max" : 394072,
"bw_agg" : 100.000000,
"bw_mean" : 392940.500000,
"bw_dev" : 744.587315,
"bw_samples" : 14,
"iops_min" : 97889,
"iops_max" : 98520,
"iops_mean" : 98235.214286,
"iops_stddev" : 186.488267,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 12.885714,
"sys_cpu" : 39.400000,
"ctx" : 343653,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010471,
"20" : 95.019649,
"50" : 4.858766,
"100" : 0.108642,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=85953: Tue Oct 25 23:31:56 2022
read: IOPS=98.2k, BW=384MiB/s (402MB/s)(2686MiB/7001msec)
slat (nsec): min=2003, max=40265, avg=2351.19, stdev=337.79
clat (usec): min=3, max=209, avg=17.50, stdev= 3.23
lat (usec): min=11, max=211, avg=19.85, stdev= 3.22
clat percentiles (nsec):
| 1.00th=[16320], 5.00th=[16512], 10.00th=[16512], 20.00th=[16512],
| 30.00th=[16512], 40.00th=[16768], 50.00th=[17024], 60.00th=[17024],
| 70.00th=[17280], 80.00th=[17280], 90.00th=[17280], 95.00th=[19584],
| 99.00th=[33536], 99.50th=[34560], 99.90th=[50432], 99.95th=[63744],
| 99.99th=[74240]
bw ( KiB/s): min=391559, max=394072, per=100.00%, avg=392940.50, stdev=744.59, samples=14
iops : min=97889, max=98520, avg=98235.21, stdev=186.49, samples=14
lat (usec) : 4=0.01%, 10=0.01%, 20=95.02%, 50=4.86%, 100=0.11%
lat (usec) : 250=0.01%
cpu : usr=12.89%, sys=39.40%, ctx=343653, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=687582,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=384MiB/s (402MB/s), 384MiB/s-384MiB/s (402MB/s-402MB/s), io=2686MiB (2816MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt"
$ cat /tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733529,
"timestamp_ms" : 1666733529910,
"time" : "Tue Oct 25 23:32:09 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 2630627328,
"io_kbytes" : 2568972,
"bw_bytes" : 375750225,
"bw" : 366943,
"iops" : 91735.752035,
"runtime" : 7001,
"total_ios" : 642242,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2124,
"max" : 14818,
"mean" : 2496.051688,
"stddev" : 247.656662,
"N" : 642242
},
"clat_ns" : {
"min" : 4859,
"max" : 136816,
"mean" : 18815.614652,
"stddev" : 3328.463438,
"N" : 642243,
"percentile" : {
"1.000000" : 17536,
"5.000000" : 17792,
"10.000000" : 17792,
"20.000000" : 17792,
"30.000000" : 17792,
"40.000000" : 18048,
"50.000000" : 18304,
"60.000000" : 18304,
"70.000000" : 18304,
"80.000000" : 18560,
"90.000000" : 18560,
"95.000000" : 21888,
"99.000000" : 35072,
"99.500000" : 44288,
"99.900000" : 49920,
"99.950000" : 62208,
"99.990000" : 74240
}
},
"lat_ns" : {
"min" : 12043,
"max" : 139101,
"mean" : 21311.665947,
"stddev" : 3318.507739,
"N" : 642243
},
"bw_min" : 366640,
"bw_max" : 368008,
"bw_agg" : 100.000000,
"bw_mean" : 367114.357143,
"bw_dev" : 409.196136,
"bw_samples" : 14,
"iops_min" : 91660,
"iops_max" : 92002,
"iops_mean" : 91778.500000,
"iops_stddev" : 102.304034,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 12.457143,
"sys_cpu" : 36.600000,
"ctx" : 321062,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 94.633643,
"50" : 5.261257,
"100" : 0.097627,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=85992: Tue Oct 25 23:32:09 2022
read: IOPS=91.7k, BW=358MiB/s (376MB/s)(2509MiB/7001msec)
slat (nsec): min=2124, max=14818, avg=2496.05, stdev=247.66
clat (usec): min=4, max=136, avg=18.82, stdev= 3.33
lat (usec): min=12, max=139, avg=21.31, stdev= 3.32
clat percentiles (nsec):
| 1.00th=[17536], 5.00th=[17792], 10.00th=[17792], 20.00th=[17792],
| 30.00th=[17792], 40.00th=[18048], 50.00th=[18304], 60.00th=[18304],
| 70.00th=[18304], 80.00th=[18560], 90.00th=[18560], 95.00th=[21888],
| 99.00th=[35072], 99.50th=[44288], 99.90th=[49920], 99.95th=[62208],
| 99.99th=[74240]
bw ( KiB/s): min=366640, max=368008, per=100.00%, avg=367114.36, stdev=409.20, samples=14
iops : min=91660, max=92002, avg=91778.50, stdev=102.30, samples=14
lat (usec) : 10=0.01%, 20=94.63%, 50=5.26%, 100=0.10%, 250=0.01%
cpu : usr=12.46%, sys=36.60%, ctx=321062, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=642242,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=358MiB/s (376MB/s), 358MiB/s-358MiB/s (376MB/s-376MB/s), io=2509MiB (2631MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt"
$ cat /tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733543,
"timestamp_ms" : 1666733543542,
"time" : "Tue Oct 25 23:32:23 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 2749571072,
"io_kbytes" : 2685128,
"bw_bytes" : 392739761,
"bw" : 383534,
"iops" : 95883.588059,
"runtime" : 7001,
"total_ios" : 671281,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2064,
"max" : 31810,
"mean" : 2400.967224,
"stddev" : 306.105301,
"N" : 671281
},
"clat_ns" : {
"min" : 391,
"max" : 135253,
"mean" : 17952.067264,
"stddev" : 3287.475107,
"N" : 671282,
"percentile" : {
"1.000000" : 16512,
"5.000000" : 16512,
"10.000000" : 16768,
"20.000000" : 16768,
"30.000000" : 16768,
"40.000000" : 17280,
"50.000000" : 17280,
"60.000000" : 17280,
"70.000000" : 17792,
"80.000000" : 18048,
"90.000000" : 18304,
"95.000000" : 20352,
"99.000000" : 34048,
"99.500000" : 37120,
"99.900000" : 50432,
"99.950000" : 63744,
"99.990000" : 74240
}
},
"lat_ns" : {
"min" : 11592,
"max" : 143409,
"mean" : 20353.034373,
"stddev" : 3289.918264,
"N" : 671282
},
"bw_min" : 367224,
"bw_max" : 390992,
"bw_agg" : 100.000000,
"bw_mean" : 383631.928571,
"bw_dev" : 9319.619474,
"bw_samples" : 14,
"iops_min" : 91806,
"iops_max" : 97748,
"iops_mean" : 95907.928571,
"iops_stddev" : 2330.003977,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.528571,
"sys_cpu" : 37.685714,
"ctx" : 335517,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.011471,
"20" : 94.923586,
"50" : 4.958132,
"100" : 0.104278,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=86031: Tue Oct 25 23:32:23 2022
read: IOPS=95.9k, BW=375MiB/s (393MB/s)(2622MiB/7001msec)
slat (nsec): min=2064, max=31810, avg=2400.97, stdev=306.11
clat (nsec): min=391, max=135253, avg=17952.07, stdev=3287.48
lat (usec): min=11, max=143, avg=20.35, stdev= 3.29
clat percentiles (nsec):
| 1.00th=[16512], 5.00th=[16512], 10.00th=[16768], 20.00th=[16768],
| 30.00th=[16768], 40.00th=[17280], 50.00th=[17280], 60.00th=[17280],
| 70.00th=[17792], 80.00th=[18048], 90.00th=[18304], 95.00th=[20352],
| 99.00th=[34048], 99.50th=[37120], 99.90th=[50432], 99.95th=[63744],
| 99.99th=[74240]
bw ( KiB/s): min=367224, max=390992, per=100.00%, avg=383631.93, stdev=9319.62, samples=14
iops : min=91806, max=97748, avg=95907.93, stdev=2330.00, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=94.92%, 50=4.96%, 100=0.10%
lat (usec) : 250=0.01%
cpu : usr=13.53%, sys=37.69%, ctx=335517, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=671281,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=375MiB/s (393MB/s), 375MiB/s-375MiB/s (393MB/s-393MB/s), io=2622MiB (2750MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733043,
"timestamp_ms" : 1666733043592,
"time" : "Tue Oct 25 23:24:03 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 1862557696,
"io_kbytes" : 1818904,
"bw_bytes" : 266041664,
"bw" : 259806,
"iops" : 64951.578346,
"runtime" : 7001,
"total_ios" : 454726,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2314,
"max" : 40516,
"mean" : 2487.807033,
"stddev" : 208.501404,
"N" : 454726
},
"clat_ns" : {
"min" : 351,
"max" : 1341115,
"mean" : 12323.750791,
"stddev" : 2811.725511,
"N" : 454726,
"percentile" : {
"1.000000" : 11840,
"5.000000" : 11840,
"10.000000" : 11840,
"20.000000" : 11840,
"30.000000" : 11968,
"40.000000" : 11968,
"50.000000" : 12096,
"60.000000" : 12096,
"70.000000" : 12096,
"80.000000" : 12224,
"90.000000" : 12224,
"95.000000" : 12480,
"99.000000" : 22144,
"99.500000" : 23680,
"99.900000" : 31616,
"99.950000" : 50432,
"99.990000" : 64256
}
},
"lat_ns" : {
"min" : 12694,
"max" : 1343630,
"mean" : 14811.557824,
"stddev" : 2820.004980,
"N" : 454726
},
"bw_min" : 259104,
"bw_max" : 260440,
"bw_agg" : 100.000000,
"bw_mean" : 259877.071429,
"bw_dev" : 357.766676,
"bw_samples" : 14,
"iops_min" : 64776,
"iops_max" : 65110,
"iops_mean" : 64969.214286,
"iops_stddev" : 89.484144,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 11.557143,
"sys_cpu" : 35.457143,
"ctx" : 454702,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 98.417728,
"50" : 1.522235,
"100" : 0.051679,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=84507: Tue Oct 25 23:24:03 2022
read: IOPS=65.0k, BW=254MiB/s (266MB/s)(1776MiB/7001msec)
slat (nsec): min=2314, max=40516, avg=2487.81, stdev=208.50
clat (nsec): min=351, max=1341.1k, avg=12323.75, stdev=2811.73
lat (usec): min=12, max=1343, avg=14.81, stdev= 2.82
clat percentiles (nsec):
| 1.00th=[11840], 5.00th=[11840], 10.00th=[11840], 20.00th=[11840],
| 30.00th=[11968], 40.00th=[11968], 50.00th=[12096], 60.00th=[12096],
| 70.00th=[12096], 80.00th=[12224], 90.00th=[12224], 95.00th=[12480],
| 99.00th=[22144], 99.50th=[23680], 99.90th=[31616], 99.95th=[50432],
| 99.99th=[64256]
bw ( KiB/s): min=259104, max=260440, per=100.00%, avg=259877.07, stdev=357.77, samples=14
iops : min=64776, max=65110, avg=64969.21, stdev=89.48, samples=14
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 10=0.01%, 20=98.42%, 50=1.52%, 100=0.05%, 250=0.01%
lat (msec) : 2=0.01%
cpu : usr=11.56%, sys=35.46%, ctx=454702, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=454726,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=254MiB/s (266MB/s), 254MiB/s-254MiB/s (266MB/s-266MB/s), io=1776MiB (1863MB), run=7001-7001msec
$ cat /tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733558,
"timestamp_ms" : 1666733558546,
"time" : "Tue Oct 25 23:32:38 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3217563648,
"io_kbytes" : 3142152,
"bw_bytes" : 459586294,
"bw" : 448814,
"iops" : 112203.542351,
"runtime" : 7001,
"total_ios" : 785537,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 661,
"max" : 7804,
"mean" : 916.427430,
"stddev" : 81.567851,
"N" : 785537
},
"clat_ns" : {
"min" : 7284,
"max" : 134732,
"mean" : 16623.412960,
"stddev" : 3660.577461,
"N" : 785538,
"percentile" : {
"1.000000" : 13632,
"5.000000" : 13888,
"10.000000" : 14016,
"20.000000" : 14144,
"30.000000" : 14272,
"40.000000" : 14400,
"50.000000" : 17280,
"60.000000" : 17536,
"70.000000" : 17792,
"80.000000" : 18048,
"90.000000" : 18048,
"95.000000" : 19584,
"99.000000" : 30848,
"99.500000" : 34560,
"99.900000" : 51456,
"99.950000" : 65280,
"99.990000" : 79360
}
},
"lat_ns" : {
"min" : 11752,
"max" : 135514,
"mean" : 17539.840385,
"stddev" : 3659.715140,
"N" : 785538
},
"bw_min" : 447328,
"bw_max" : 451014,
"bw_agg" : 100.000000,
"bw_mean" : 448941.000000,
"bw_dev" : 1118.741257,
"bw_samples" : 14,
"iops_min" : 111832,
"iops_max" : 112753,
"iops_mean" : 112235.357143,
"iops_stddev" : 279.439991,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 2,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 99.983196,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 95.054466,
"50" : 4.836819,
"100" : 0.108334,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=86080: Tue Oct 25 23:32:38 2022
read: IOPS=112k, BW=438MiB/s (460MB/s)(3069MiB/7001msec)
slat (nsec): min=661, max=7804, avg=916.43, stdev=81.57
clat (usec): min=7, max=134, avg=16.62, stdev= 3.66
lat (usec): min=11, max=135, avg=17.54, stdev= 3.66
clat percentiles (nsec):
| 1.00th=[13632], 5.00th=[13888], 10.00th=[14016], 20.00th=[14144],
| 30.00th=[14272], 40.00th=[14400], 50.00th=[17280], 60.00th=[17536],
| 70.00th=[17792], 80.00th=[18048], 90.00th=[18048], 95.00th=[19584],
| 99.00th=[30848], 99.50th=[34560], 99.90th=[51456], 99.95th=[65280],
| 99.99th=[79360]
bw ( KiB/s): min=447328, max=451014, per=100.00%, avg=448941.00, stdev=1118.74, samples=14
iops : min=111832, max=112753, avg=112235.36, stdev=279.44, samples=14
lat (usec) : 10=0.01%, 20=95.05%, 50=4.84%, 100=0.11%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=2, majf=0, minf=0
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=785537,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=438MiB/s (460MB/s), 438MiB/s-438MiB/s (460MB/s-460MB/s), io=3069MiB (3218MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733573,
"timestamp_ms" : 1666733573691,
"time" : "Tue Oct 25 23:32:53 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3401510912,
"io_kbytes" : 3321788,
"bw_bytes" : 485860721,
"bw" : 474473,
"iops" : 118618.197400,
"runtime" : 7001,
"total_ios" : 830446,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 671,
"max" : 11402,
"mean" : 967.528109,
"stddev" : 130.573892,
"N" : 830446
},
"clat_ns" : {
"min" : 7574,
"max" : 176641,
"mean" : 15406.331199,
"stddev" : 3395.306415,
"N" : 830447,
"percentile" : {
"1.000000" : 13632,
"5.000000" : 13760,
"10.000000" : 13888,
"20.000000" : 14656,
"30.000000" : 14784,
"40.000000" : 14784,
"50.000000" : 14912,
"60.000000" : 15040,
"70.000000" : 15040,
"80.000000" : 15168,
"90.000000" : 15296,
"95.000000" : 15808,
"99.000000" : 31104,
"99.500000" : 36096,
"99.900000" : 51456,
"99.950000" : 65280,
"99.990000" : 82432
}
},
"lat_ns" : {
"min" : 10650,
"max" : 177563,
"mean" : 16373.859542,
"stddev" : 3397.897104,
"N" : 830447
},
"bw_min" : 473232,
"bw_max" : 475744,
"bw_agg" : 100.000000,
"bw_mean" : 474607.785714,
"bw_dev" : 803.230801,
"bw_samples" : 14,
"iops_min" : 118308,
"iops_max" : 118936,
"iops_mean" : 118651.928571,
"iops_stddev" : 200.807778,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 7,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 15.182565,
"2" : 84.817435,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 95.736147,
"50" : 4.161378,
"100" : 0.099224,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=86130: Tue Oct 25 23:32:53 2022
read: IOPS=119k, BW=463MiB/s (486MB/s)(3244MiB/7001msec)
slat (nsec): min=671, max=11402, avg=967.53, stdev=130.57
clat (usec): min=7, max=176, avg=15.41, stdev= 3.40
lat (usec): min=10, max=177, avg=16.37, stdev= 3.40
clat percentiles (nsec):
| 1.00th=[13632], 5.00th=[13760], 10.00th=[13888], 20.00th=[14656],
| 30.00th=[14784], 40.00th=[14784], 50.00th=[14912], 60.00th=[15040],
| 70.00th=[15040], 80.00th=[15168], 90.00th=[15296], 95.00th=[15808],
| 99.00th=[31104], 99.50th=[36096], 99.90th=[51456], 99.95th=[65280],
| 99.99th=[82432]
bw ( KiB/s): min=473232, max=475744, per=100.00%, avg=474607.79, stdev=803.23, samples=14
iops : min=118308, max=118936, avg=118651.93, stdev=200.81, samples=14
lat (usec) : 10=0.01%, 20=95.74%, 50=4.16%, 100=0.10%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=7, majf=0, minf=0
IO depths : 1=15.2%, 2=84.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=830446,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=463MiB/s (486MB/s), 463MiB/s-463MiB/s (486MB/s-486MB/s), io=3244MiB (3402MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733588,
"timestamp_ms" : 1666733588818,
"time" : "Tue Oct 25 23:33:08 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3410411520,
"io_kbytes" : 3330480,
"bw_bytes" : 487132055,
"bw" : 475714,
"iops" : 118928.581631,
"runtime" : 7001,
"total_ios" : 832619,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 661,
"max" : 50916,
"mean" : 946.325844,
"stddev" : 169.673091,
"N" : 832619
},
"clat_ns" : {
"min" : 881,
"max" : 1304386,
"mean" : 15275.264800,
"stddev" : 3935.692281,
"N" : 832620,
"percentile" : {
"1.000000" : 13504,
"5.000000" : 13632,
"10.000000" : 13760,
"20.000000" : 13888,
"30.000000" : 14656,
"40.000000" : 14784,
"50.000000" : 14784,
"60.000000" : 14912,
"70.000000" : 15040,
"80.000000" : 15040,
"90.000000" : 15296,
"95.000000" : 15808,
"99.000000" : 31104,
"99.500000" : 36608,
"99.900000" : 48896,
"99.950000" : 64768,
"99.990000" : 82432
}
},
"lat_ns" : {
"min" : 10650,
"max" : 1305568,
"mean" : 16221.590603,
"stddev" : 3939.658232,
"N" : 832620
},
"bw_min" : 474520,
"bw_max" : 477072,
"bw_agg" : 100.000000,
"bw_mean" : 475836.000000,
"bw_dev" : 739.461759,
"bw_samples" : 14,
"iops_min" : 118630,
"iops_max" : 119268,
"iops_mean" : 118959.000000,
"iops_stddev" : 184.865440,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 19,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 25.146916,
"2" : 74.853084,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 95.767812,
"50" : 4.132622,
"100" : 0.094521,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=86180: Tue Oct 25 23:33:08 2022
read: IOPS=119k, BW=465MiB/s (487MB/s)(3252MiB/7001msec)
slat (nsec): min=661, max=50916, avg=946.33, stdev=169.67
clat (nsec): min=881, max=1304.4k, avg=15275.26, stdev=3935.69
lat (usec): min=10, max=1305, avg=16.22, stdev= 3.94
clat percentiles (nsec):
| 1.00th=[13504], 5.00th=[13632], 10.00th=[13760], 20.00th=[13888],
| 30.00th=[14656], 40.00th=[14784], 50.00th=[14784], 60.00th=[14912],
| 70.00th=[15040], 80.00th=[15040], 90.00th=[15296], 95.00th=[15808],
| 99.00th=[31104], 99.50th=[36608], 99.90th=[48896], 99.95th=[64768],
| 99.99th=[82432]
bw ( KiB/s): min=474520, max=477072, per=100.00%, avg=475836.00, stdev=739.46, samples=14
iops : min=118630, max=119268, avg=118959.00, stdev=184.87, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=95.77%, 50=4.13%
lat (usec) : 100=0.09%, 250=0.01%
lat (msec) : 2=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=19, majf=0, minf=0
IO depths : 1=25.1%, 2=74.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=832619,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=465MiB/s (487MB/s), 465MiB/s-465MiB/s (487MB/s-487MB/s), io=3252MiB (3410MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-reference_0.txt"
$ cat /tmp/fio-output_4K_2_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733602,
"timestamp_ms" : 1666733602544,
"time" : "Tue Oct 25 23:33:22 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 2728185856,
"io_kbytes" : 2664244,
"bw_bytes" : 389685167,
"bw" : 380551,
"iops" : 95137.837452,
"runtime" : 7001,
"total_ios" : 666060,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2254,
"max" : 101570,
"mean" : 2532.068260,
"stddev" : 388.305482,
"N" : 666060
},
"clat_ns" : {
"min" : 150,
"max" : 157766,
"mean" : 17905.455248,
"stddev" : 3291.383779,
"N" : 666061,
"percentile" : {
"1.000000" : 16768,
"5.000000" : 16768,
"10.000000" : 16768,
"20.000000" : 17024,
"30.000000" : 17024,
"40.000000" : 17024,
"50.000000" : 17280,
"60.000000" : 17536,
"70.000000" : 17536,
"80.000000" : 17536,
"90.000000" : 17792,
"95.000000" : 19840,
"99.000000" : 34048,
"99.500000" : 40192,
"99.900000" : 50432,
"99.950000" : 63232,
"99.990000" : 74240
}
},
"lat_ns" : {
"min" : 11642,
"max" : 160081,
"mean" : 20437.523182,
"stddev" : 3295.392638,
"N" : 666061
},
"bw_min" : 379696,
"bw_max" : 382128,
"bw_agg" : 100.000000,
"bw_mean" : 380707.500000,
"bw_dev" : 705.250065,
"bw_samples" : 14,
"iops_min" : 94924,
"iops_max" : 95532,
"iops_mean" : 95176.857143,
"iops_stddev" : 176.305180,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 12.700000,
"sys_cpu" : 41.028571,
"ctx" : 332854,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.016215,
"20" : 95.018317,
"50" : 4.859622,
"100" : 0.102093,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 965358,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 12002,
"write_ticks" : 0,
"in_queue" : 12002,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=86219: Tue Oct 25 23:33:22 2022
read: IOPS=95.1k, BW=372MiB/s (390MB/s)(2602MiB/7001msec)
slat (nsec): min=2254, max=101570, avg=2532.07, stdev=388.31
clat (nsec): min=150, max=157766, avg=17905.46, stdev=3291.38
lat (usec): min=11, max=160, avg=20.44, stdev= 3.30
clat percentiles (nsec):
| 1.00th=[16768], 5.00th=[16768], 10.00th=[16768], 20.00th=[17024],
| 30.00th=[17024], 40.00th=[17024], 50.00th=[17280], 60.00th=[17536],
| 70.00th=[17536], 80.00th=[17536], 90.00th=[17792], 95.00th=[19840],
| 99.00th=[34048], 99.50th=[40192], 99.90th=[50432], 99.95th=[63232],
| 99.99th=[74240]
bw ( KiB/s): min=379696, max=382128, per=100.00%, avg=380707.50, stdev=705.25, samples=14
iops : min=94924, max=95532, avg=95176.86, stdev=176.31, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=0.02%, 20=95.02%, 50=4.86%, 100=0.10%
lat (usec) : 250=0.01%
cpu : usr=12.70%, sys=41.03%, ctx=332854, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=666060,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=372MiB/s (390MB/s), 372MiB/s-372MiB/s (390MB/s-390MB/s), io=2602MiB (2728MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=965358/0, merge=0/0, ticks=12002/0, in_queue=12002, util=99.09%
$ rm /tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt': No such file or directory
$ rm /tmp/fio-output_4K_2_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-reference_1.txt"
$ cat /tmp/fio-output_4K_2_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733616,
"timestamp_ms" : 1666733616144,
"time" : "Tue Oct 25 23:33:36 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 2763063296,
"io_kbytes" : 2698304,
"bw_bytes" : 394666947,
"bw" : 385416,
"iops" : 96354.092273,
"runtime" : 7001,
"total_ios" : 674575,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2134,
"max" : 30237,
"mean" : 2411.264627,
"stddev" : 301.369070,
"N" : 674575
},
"clat_ns" : {
"min" : 3416,
"max" : 152055,
"mean" : 17806.873363,
"stddev" : 3321.266952,
"N" : 674576,
"percentile" : {
"1.000000" : 16768,
"5.000000" : 16768,
"10.000000" : 16768,
"20.000000" : 16768,
"30.000000" : 17024,
"40.000000" : 17024,
"50.000000" : 17280,
"60.000000" : 17280,
"70.000000" : 17536,
"80.000000" : 17536,
"90.000000" : 17536,
"95.000000" : 20096,
"99.000000" : 34048,
"99.500000" : 40192,
"99.900000" : 53504,
"99.950000" : 65280,
"99.990000" : 76288
}
},
"lat_ns" : {
"min" : 11702,
"max" : 154269,
"mean" : 20218.137700,
"stddev" : 3319.032476,
"N" : 674576
},
"bw_min" : 384640,
"bw_max" : 386432,
"bw_agg" : 100.000000,
"bw_mean" : 385564.928571,
"bw_dev" : 441.118241,
"bw_samples" : 14,
"iops_min" : 96160,
"iops_max" : 96608,
"iops_mean" : 96391.142857,
"iops_stddev" : 110.190644,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 11.157143,
"sys_cpu" : 41.785714,
"ctx" : 337155,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 94.990624,
"50" : 4.873884,
"100" : 0.124226,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 966533,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 11927,
"write_ticks" : 0,
"in_queue" : 11927,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=86257: Tue Oct 25 23:33:36 2022
read: IOPS=96.4k, BW=376MiB/s (395MB/s)(2635MiB/7001msec)
slat (nsec): min=2134, max=30237, avg=2411.26, stdev=301.37
clat (usec): min=3, max=152, avg=17.81, stdev= 3.32
lat (usec): min=11, max=154, avg=20.22, stdev= 3.32
clat percentiles (nsec):
| 1.00th=[16768], 5.00th=[16768], 10.00th=[16768], 20.00th=[16768],
| 30.00th=[17024], 40.00th=[17024], 50.00th=[17280], 60.00th=[17280],
| 70.00th=[17536], 80.00th=[17536], 90.00th=[17536], 95.00th=[20096],
| 99.00th=[34048], 99.50th=[40192], 99.90th=[53504], 99.95th=[65280],
| 99.99th=[76288]
bw ( KiB/s): min=384640, max=386432, per=100.00%, avg=385564.93, stdev=441.12, samples=14
iops : min=96160, max=96608, avg=96391.14, stdev=110.19, samples=14
lat (usec) : 4=0.01%, 10=0.01%, 20=94.99%, 50=4.87%, 100=0.12%
lat (usec) : 250=0.01%
cpu : usr=11.16%, sys=41.79%, ctx=337155, majf=0, minf=1
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=674575,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=376MiB/s (395MB/s), 376MiB/s-376MiB/s (395MB/s-395MB/s), io=2635MiB (2763MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=966533/0, merge=0/0, ticks=11927/0, in_queue=11927, util=99.09%
$ rm /tmp/fio-output_4K_2_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-reference_2.txt"
$ cat /tmp/fio-output_4K_2_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733629,
"timestamp_ms" : 1666733629769,
"time" : "Tue Oct 25 23:33:49 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 2769817600,
"io_kbytes" : 2704900,
"bw_bytes" : 395631709,
"bw" : 386359,
"iops" : 96589.630053,
"runtime" : 7001,
"total_ios" : 676224,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2174,
"max" : 56947,
"mean" : 2455.820520,
"stddev" : 313.047630,
"N" : 676224
},
"clat_ns" : {
"min" : 3426,
"max" : 121939,
"mean" : 17701.656814,
"stddev" : 3232.617042,
"N" : 676225,
"percentile" : {
"1.000000" : 16512,
"5.000000" : 16768,
"10.000000" : 16768,
"20.000000" : 16768,
"30.000000" : 16768,
"40.000000" : 16768,
"50.000000" : 17280,
"60.000000" : 17280,
"70.000000" : 17280,
"80.000000" : 17280,
"90.000000" : 17536,
"95.000000" : 19840,
"99.000000" : 34048,
"99.500000" : 36608,
"99.900000" : 50432,
"99.950000" : 63744,
"99.990000" : 74240
}
},
"lat_ns" : {
"min" : 11822,
"max" : 124183,
"mean" : 20157.477096,
"stddev" : 3230.430762,
"N" : 676225
},
"bw_min" : 385176,
"bw_max" : 387294,
"bw_agg" : 100.000000,
"bw_mean" : 386515.642857,
"bw_dev" : 702.422359,
"bw_samples" : 14,
"iops_min" : 96294,
"iops_max" : 96823,
"iops_mean" : 96628.857143,
"iops_stddev" : 175.563207,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 10.471429,
"sys_cpu" : 42.771429,
"ctx" : 337987,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 94.997516,
"50" : 4.886694,
"100" : 0.107509,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 967584,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 11897,
"write_ticks" : 0,
"in_queue" : 11897,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=86295: Tue Oct 25 23:33:49 2022
read: IOPS=96.6k, BW=377MiB/s (396MB/s)(2642MiB/7001msec)
slat (nsec): min=2174, max=56947, avg=2455.82, stdev=313.05
clat (usec): min=3, max=121, avg=17.70, stdev= 3.23
lat (usec): min=11, max=124, avg=20.16, stdev= 3.23
clat percentiles (nsec):
| 1.00th=[16512], 5.00th=[16768], 10.00th=[16768], 20.00th=[16768],
| 30.00th=[16768], 40.00th=[16768], 50.00th=[17280], 60.00th=[17280],
| 70.00th=[17280], 80.00th=[17280], 90.00th=[17536], 95.00th=[19840],
| 99.00th=[34048], 99.50th=[36608], 99.90th=[50432], 99.95th=[63744],
| 99.99th=[74240]
bw ( KiB/s): min=385176, max=387294, per=100.00%, avg=386515.64, stdev=702.42, samples=14
iops : min=96294, max=96823, avg=96628.86, stdev=175.56, samples=14
lat (usec) : 4=0.01%, 10=0.01%, 20=95.00%, 50=4.89%, 100=0.11%
lat (usec) : 250=0.01%
cpu : usr=10.47%, sys=42.77%, ctx=337987, majf=0, minf=1
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=676224,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=377MiB/s (396MB/s), 377MiB/s-377MiB/s (396MB/s-396MB/s), io=2642MiB (2770MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=967584/0, merge=0/0, ticks=11897/0, in_queue=11897, util=99.09%
$ rm /tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733644,
"timestamp_ms" : 1666733644623,
"time" : "Tue Oct 25 23:34:04 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3403218944,
"io_kbytes" : 3323456,
"bw_bytes" : 486104691,
"bw" : 474711,
"iops" : 118677.760320,
"runtime" : 7001,
"total_ios" : 830863,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 13305,
"mean" : 152.035344,
"stddev" : 44.246764,
"N" : 830863
},
"clat_ns" : {
"min" : 11011,
"max" : 125345,
"mean" : 16229.591323,
"stddev" : 3439.794272,
"N" : 830863,
"percentile" : {
"1.000000" : 15168,
"5.000000" : 15168,
"10.000000" : 15296,
"20.000000" : 15296,
"30.000000" : 15424,
"40.000000" : 15552,
"50.000000" : 15552,
"60.000000" : 15680,
"70.000000" : 15680,
"80.000000" : 15808,
"90.000000" : 15936,
"95.000000" : 16512,
"99.000000" : 31872,
"99.500000" : 37632,
"99.900000" : 49920,
"99.950000" : 66048,
"99.990000" : 84480
}
},
"lat_ns" : {
"min" : 11191,
"max" : 125525,
"mean" : 16381.626666,
"stddev" : 3441.514411,
"N" : 830863
},
"bw_min" : 473656,
"bw_max" : 476112,
"bw_agg" : 100.000000,
"bw_mean" : 474806.214286,
"bw_dev" : 872.970718,
"bw_samples" : 14,
"iops_min" : 118414,
"iops_max" : 119028,
"iops_mean" : 118701.214286,
"iops_stddev" : 218.461818,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 77.185714,
"sys_cpu" : 22.785714,
"ctx" : 16,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 49.969128,
"2" : 50.030872,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 95.630567,
"50" : 4.272425,
"100" : 0.096526,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=86344: Tue Oct 25 23:34:04 2022
read: IOPS=119k, BW=464MiB/s (486MB/s)(3246MiB/7001msec)
slat (nsec): min=110, max=13305, avg=152.04, stdev=44.25
clat (usec): min=11, max=125, avg=16.23, stdev= 3.44
lat (usec): min=11, max=125, avg=16.38, stdev= 3.44
clat percentiles (nsec):
| 1.00th=[15168], 5.00th=[15168], 10.00th=[15296], 20.00th=[15296],
| 30.00th=[15424], 40.00th=[15552], 50.00th=[15552], 60.00th=[15680],
| 70.00th=[15680], 80.00th=[15808], 90.00th=[15936], 95.00th=[16512],
| 99.00th=[31872], 99.50th=[37632], 99.90th=[49920], 99.95th=[66048],
| 99.99th=[84480]
bw ( KiB/s): min=473656, max=476112, per=100.00%, avg=474806.21, stdev=872.97, samples=14
iops : min=118414, max=119028, avg=118701.21, stdev=218.46, samples=14
lat (usec) : 20=95.63%, 50=4.27%, 100=0.10%, 250=0.01%
cpu : usr=77.19%, sys=22.79%, ctx=16, majf=0, minf=0
IO depths : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=830863,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=464MiB/s (486MB/s), 464MiB/s-464MiB/s (486MB/s-486MB/s), io=3246MiB (3403MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt"
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733659,
"timestamp_ms" : 1666733659767,
"time" : "Tue Oct 25 23:34:19 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3699757056,
"io_kbytes" : 3613044,
"bw_bytes" : 528461227,
"bw" : 516075,
"iops" : 129018.711613,
"runtime" : 7001,
"total_ios" : 903260,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 13215,
"mean" : 159.408986,
"stddev" : 56.703624,
"N" : 903260
},
"clat_ns" : {
"min" : 9217,
"max" : 1316439,
"mean" : 14838.287266,
"stddev" : 4093.044611,
"N" : 903260,
"percentile" : {
"1.000000" : 13888,
"5.000000" : 13888,
"10.000000" : 13888,
"20.000000" : 14016,
"30.000000" : 14016,
"40.000000" : 14144,
"50.000000" : 14144,
"60.000000" : 14272,
"70.000000" : 14272,
"80.000000" : 14400,
"90.000000" : 14400,
"95.000000" : 15168,
"99.000000" : 31616,
"99.500000" : 39168,
"99.900000" : 48384,
"99.950000" : 66048,
"99.990000" : 84480
}
},
"lat_ns" : {
"min" : 9407,
"max" : 1316629,
"mean" : 14997.696252,
"stddev" : 4094.615509,
"N" : 903260
},
"bw_min" : 514352,
"bw_max" : 517672,
"bw_agg" : 100.000000,
"bw_mean" : 516195.357143,
"bw_dev" : 856.680679,
"bw_samples" : 14,
"iops_min" : 128588,
"iops_max" : 129418,
"iops_mean" : 129048.785714,
"iops_stddev" : 214.201264,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 68.600000,
"sys_cpu" : 31.357143,
"ctx" : 50,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 49.986936,
"2" : 50.013064,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.020924,
"20" : 96.007240,
"50" : 3.875518,
"100" : 0.095654,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=86393: Tue Oct 25 23:34:19 2022
read: IOPS=129k, BW=504MiB/s (528MB/s)(3528MiB/7001msec)
slat (nsec): min=110, max=13215, avg=159.41, stdev=56.70
clat (usec): min=9, max=1316, avg=14.84, stdev= 4.09
lat (usec): min=9, max=1316, avg=15.00, stdev= 4.09
clat percentiles (nsec):
| 1.00th=[13888], 5.00th=[13888], 10.00th=[13888], 20.00th=[14016],
| 30.00th=[14016], 40.00th=[14144], 50.00th=[14144], 60.00th=[14272],
| 70.00th=[14272], 80.00th=[14400], 90.00th=[14400], 95.00th=[15168],
| 99.00th=[31616], 99.50th=[39168], 99.90th=[48384], 99.95th=[66048],
| 99.99th=[84480]
bw ( KiB/s): min=514352, max=517672, per=100.00%, avg=516195.36, stdev=856.68, samples=14
iops : min=128588, max=129418, avg=129048.79, stdev=214.20, samples=14
lat (usec) : 10=0.02%, 20=96.01%, 50=3.88%, 100=0.10%, 250=0.01%
lat (msec) : 2=0.01%
cpu : usr=68.60%, sys=31.36%, ctx=50, majf=0, minf=0
IO depths : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=903260,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=504MiB/s (528MB/s), 504MiB/s-504MiB/s (528MB/s-528MB/s), io=3528MiB (3700MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733674,
"timestamp_ms" : 1666733674905,
"time" : "Tue Oct 25 23:34:34 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3672137728,
"io_kbytes" : 3586072,
"bw_bytes" : 524516173,
"bw" : 512222,
"iops" : 128055.563491,
"runtime" : 7001,
"total_ios" : 896517,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 100,
"max" : 16821,
"mean" : 151.655918,
"stddev" : 54.676795,
"N" : 896517
},
"clat_ns" : {
"min" : 9408,
"max" : 102191,
"mean" : 14971.500404,
"stddev" : 3599.731107,
"N" : 896517,
"percentile" : {
"1.000000" : 14016,
"5.000000" : 14016,
"10.000000" : 14144,
"20.000000" : 14144,
"30.000000" : 14144,
"40.000000" : 14272,
"50.000000" : 14272,
"60.000000" : 14400,
"70.000000" : 14400,
"80.000000" : 14400,
"90.000000" : 14528,
"95.000000" : 15296,
"99.000000" : 31616,
"99.500000" : 39168,
"99.900000" : 53504,
"99.950000" : 67072,
"99.990000" : 82432
}
},
"lat_ns" : {
"min" : 9578,
"max" : 102372,
"mean" : 15123.156323,
"stddev" : 3601.418454,
"N" : 896517
},
"bw_min" : 511288,
"bw_max" : 513200,
"bw_agg" : 100.000000,
"bw_mean" : 512352.071429,
"bw_dev" : 676.281287,
"bw_samples" : 14,
"iops_min" : 127820,
"iops_max" : 128300,
"iops_mean" : 128088.000000,
"iops_stddev" : 169.357156,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 64.900000,
"sys_cpu" : 35.071429,
"ctx" : 16,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 49.996933,
"2" : 50.003067,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 95.756801,
"50" : 4.139018,
"100" : 0.101950,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=86443: Tue Oct 25 23:34:34 2022
read: IOPS=128k, BW=500MiB/s (525MB/s)(3502MiB/7001msec)
slat (nsec): min=100, max=16821, avg=151.66, stdev=54.68
clat (usec): min=9, max=102, avg=14.97, stdev= 3.60
lat (usec): min=9, max=102, avg=15.12, stdev= 3.60
clat percentiles (nsec):
| 1.00th=[14016], 5.00th=[14016], 10.00th=[14144], 20.00th=[14144],
| 30.00th=[14144], 40.00th=[14272], 50.00th=[14272], 60.00th=[14400],
| 70.00th=[14400], 80.00th=[14400], 90.00th=[14528], 95.00th=[15296],
| 99.00th=[31616], 99.50th=[39168], 99.90th=[53504], 99.95th=[67072],
| 99.99th=[82432]
bw ( KiB/s): min=511288, max=513200, per=100.00%, avg=512352.07, stdev=676.28, samples=14
iops : min=127820, max=128300, avg=128088.00, stdev=169.36, samples=14
lat (usec) : 10=0.01%, 20=95.76%, 50=4.14%, 100=0.10%, 250=0.01%
cpu : usr=64.90%, sys=35.07%, ctx=16, majf=0, minf=0
IO depths : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=896517,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=500MiB/s (525MB/s), 500MiB/s-500MiB/s (525MB/s-525MB/s), io=3502MiB (3672MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733689,
"timestamp_ms" : 1666733689991,
"time" : "Tue Oct 25 23:34:49 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3760844800,
"io_kbytes" : 3672700,
"bw_bytes" : 537186801,
"bw" : 524596,
"iops" : 131148.978717,
"runtime" : 7001,
"total_ios" : 918174,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1643,
"max" : 15720,
"mean" : 1814.378433,
"stddev" : 122.167456,
"N" : 918174
},
"clat_ns" : {
"min" : 261,
"max" : 95289,
"mean" : 13203.321858,
"stddev" : 4291.070630,
"N" : 918175,
"percentile" : {
"1.000000" : 10048,
"5.000000" : 10176,
"10.000000" : 10176,
"20.000000" : 10176,
"30.000000" : 10304,
"40.000000" : 10304,
"50.000000" : 14528,
"60.000000" : 14656,
"70.000000" : 14656,
"80.000000" : 14784,
"90.000000" : 14912,
"95.000000" : 15552,
"99.000000" : 31616,
"99.500000" : 37120,
"99.900000" : 44800,
"99.950000" : 62720,
"99.990000" : 76288
}
},
"lat_ns" : {
"min" : 9618,
"max" : 97092,
"mean" : 15017.700203,
"stddev" : 4292.850203,
"N" : 918175
},
"bw_min" : 523296,
"bw_max" : 526613,
"bw_agg" : 100.000000,
"bw_mean" : 524883.928571,
"bw_dev" : 955.591357,
"bw_samples" : 14,
"iops_min" : 130824,
"iops_max" : 131653,
"iops_mean" : 131220.928571,
"iops_stddev" : 238.846737,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 11.700000,
"sys_cpu" : 88.271429,
"ctx" : 13,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.690174,
"20" : 95.137632,
"50" : 4.081253,
"100" : 0.088110,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=86494: Tue Oct 25 23:34:49 2022
read: IOPS=131k, BW=512MiB/s (537MB/s)(3587MiB/7001msec)
slat (nsec): min=1643, max=15720, avg=1814.38, stdev=122.17
clat (nsec): min=261, max=95289, avg=13203.32, stdev=4291.07
lat (nsec): min=9618, max=97092, avg=15017.70, stdev=4292.85
clat percentiles (nsec):
| 1.00th=[10048], 5.00th=[10176], 10.00th=[10176], 20.00th=[10176],
| 30.00th=[10304], 40.00th=[10304], 50.00th=[14528], 60.00th=[14656],
| 70.00th=[14656], 80.00th=[14784], 90.00th=[14912], 95.00th=[15552],
| 99.00th=[31616], 99.50th=[37120], 99.90th=[44800], 99.95th=[62720],
| 99.99th=[76288]
bw ( KiB/s): min=523296, max=526613, per=100.00%, avg=524883.93, stdev=955.59, samples=14
iops : min=130824, max=131653, avg=131220.93, stdev=238.85, samples=14
lat (nsec) : 500=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.69%, 20=95.14%, 50=4.08%
lat (usec) : 100=0.09%
cpu : usr=11.70%, sys=88.27%, ctx=13, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=918174,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=512MiB/s (537MB/s), 512MiB/s-512MiB/s (537MB/s-537MB/s), io=3587MiB (3761MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733057,
"timestamp_ms" : 1666733057326,
"time" : "Tue Oct 25 23:24:17 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 1745297408,
"io_kbytes" : 1704392,
"bw_bytes" : 249292587,
"bw" : 243449,
"iops" : 60862.448222,
"runtime" : 7001,
"total_ios" : 426098,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2294,
"max" : 44142,
"mean" : 2645.758187,
"stddev" : 205.940659,
"N" : 426098
},
"clat_ns" : {
"min" : 351,
"max" : 105057,
"mean" : 13250.876676,
"stddev" : 2201.080057,
"N" : 426098,
"percentile" : {
"1.000000" : 12096,
"5.000000" : 12096,
"10.000000" : 12096,
"20.000000" : 12224,
"30.000000" : 12352,
"40.000000" : 13248,
"50.000000" : 13376,
"60.000000" : 13376,
"70.000000" : 13504,
"80.000000" : 13504,
"90.000000" : 13504,
"95.000000" : 13632,
"99.000000" : 22912,
"99.500000" : 29312,
"99.900000" : 38656,
"99.950000" : 52992,
"99.990000" : 69120
}
},
"lat_ns" : {
"min" : 14187,
"max" : 114945,
"mean" : 15896.634863,
"stddev" : 2223.085816,
"N" : 426098
},
"bw_min" : 235512,
"bw_max" : 257448,
"bw_agg" : 100.000000,
"bw_mean" : 243501.285714,
"bw_dev" : 9024.250392,
"bw_samples" : 14,
"iops_min" : 58878,
"iops_max" : 64362,
"iops_mean" : 60875.285714,
"iops_stddev" : 2256.088870,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 9.300000,
"sys_cpu" : 34.514286,
"ctx" : 426091,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 98.344982,
"50" : 1.589306,
"100" : 0.061488,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=84545: Tue Oct 25 23:24:17 2022
read: IOPS=60.9k, BW=238MiB/s (249MB/s)(1664MiB/7001msec)
slat (nsec): min=2294, max=44142, avg=2645.76, stdev=205.94
clat (nsec): min=351, max=105057, avg=13250.88, stdev=2201.08
lat (usec): min=14, max=114, avg=15.90, stdev= 2.22
clat percentiles (nsec):
| 1.00th=[12096], 5.00th=[12096], 10.00th=[12096], 20.00th=[12224],
| 30.00th=[12352], 40.00th=[13248], 50.00th=[13376], 60.00th=[13376],
| 70.00th=[13504], 80.00th=[13504], 90.00th=[13504], 95.00th=[13632],
| 99.00th=[22912], 99.50th=[29312], 99.90th=[38656], 99.95th=[52992],
| 99.99th=[69120]
bw ( KiB/s): min=235512, max=257448, per=100.00%, avg=243501.29, stdev=9024.25, samples=14
iops : min=58878, max=64362, avg=60875.29, stdev=2256.09, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 10=0.01%, 20=98.34%, 50=1.59%, 100=0.06%, 250=0.01%
cpu : usr=9.30%, sys=34.51%, ctx=426091, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=426098,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=238MiB/s (249MB/s), 238MiB/s-238MiB/s (249MB/s-249MB/s), io=1664MiB (1745MB), run=7001-7001msec
$ cat /tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733705,
"timestamp_ms" : 1666733705068,
"time" : "Tue Oct 25 23:35:05 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3756642304,
"io_kbytes" : 3668596,
"bw_bytes" : 536586531,
"bw" : 524010,
"iops" : 131002.428225,
"runtime" : 7001,
"total_ios" : 917148,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1643,
"max" : 19396,
"mean" : 1811.534450,
"stddev" : 120.874810,
"N" : 917148
},
"clat_ns" : {
"min" : 591,
"max" : 102302,
"mean" : 13208.573923,
"stddev" : 4374.320123,
"N" : 917149,
"percentile" : {
"1.000000" : 10048,
"5.000000" : 10176,
"10.000000" : 10176,
"20.000000" : 10176,
"30.000000" : 10304,
"40.000000" : 10304,
"50.000000" : 14528,
"60.000000" : 14656,
"70.000000" : 14656,
"80.000000" : 14784,
"90.000000" : 14912,
"95.000000" : 15552,
"99.000000" : 31872,
"99.500000" : 37120,
"99.900000" : 50432,
"99.950000" : 65280,
"99.990000" : 77312
}
},
"lat_ns" : {
"min" : 9618,
"max" : 104076,
"mean" : 15020.108310,
"stddev" : 4377.112921,
"N" : 917149
},
"bw_min" : 522248,
"bw_max" : 526256,
"bw_agg" : 100.000000,
"bw_mean" : 524297.500000,
"bw_dev" : 1261.415673,
"bw_samples" : 14,
"iops_min" : 130562,
"iops_max" : 131564,
"iops_mean" : 131074.214286,
"iops_stddev" : 315.322101,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 10.914286,
"sys_cpu" : 89.071429,
"ctx" : 17,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.643953,
"20" : 95.156507,
"50" : 4.090834,
"100" : 0.104999,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=86544: Tue Oct 25 23:35:05 2022
read: IOPS=131k, BW=512MiB/s (537MB/s)(3583MiB/7001msec)
slat (nsec): min=1643, max=19396, avg=1811.53, stdev=120.87
clat (nsec): min=591, max=102302, avg=13208.57, stdev=4374.32
lat (usec): min=9, max=104, avg=15.02, stdev= 4.38
clat percentiles (nsec):
| 1.00th=[10048], 5.00th=[10176], 10.00th=[10176], 20.00th=[10176],
| 30.00th=[10304], 40.00th=[10304], 50.00th=[14528], 60.00th=[14656],
| 70.00th=[14656], 80.00th=[14784], 90.00th=[14912], 95.00th=[15552],
| 99.00th=[31872], 99.50th=[37120], 99.90th=[50432], 99.95th=[65280],
| 99.99th=[77312]
bw ( KiB/s): min=522248, max=526256, per=100.00%, avg=524297.50, stdev=1261.42, samples=14
iops : min=130562, max=131564, avg=131074.21, stdev=315.32, samples=14
lat (nsec) : 750=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.64%, 20=95.16%, 50=4.09%
lat (usec) : 100=0.10%, 250=0.01%
cpu : usr=10.91%, sys=89.07%, ctx=17, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=917148,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=512MiB/s (537MB/s), 512MiB/s-512MiB/s (537MB/s-537MB/s), io=3583MiB (3757MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733720,
"timestamp_ms" : 1666733720129,
"time" : "Tue Oct 25 23:35:20 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3880730624,
"io_kbytes" : 3789776,
"bw_bytes" : 554310901,
"bw" : 541319,
"iops" : 135329.667190,
"runtime" : 7001,
"total_ios" : 947443,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1492,
"max" : 15409,
"mean" : 1579.608812,
"stddev" : 127.933694,
"N" : 947443
},
"clat_ns" : {
"min" : 270,
"max" : 118652,
"mean" : 12956.637074,
"stddev" : 3894.360504,
"N" : 947444,
"percentile" : {
"1.000000" : 10048,
"5.000000" : 10176,
"10.000000" : 10176,
"20.000000" : 11968,
"30.000000" : 12096,
"40.000000" : 12224,
"50.000000" : 12224,
"60.000000" : 12352,
"70.000000" : 12352,
"80.000000" : 14144,
"90.000000" : 14400,
"95.000000" : 14656,
"99.000000" : 30592,
"99.500000" : 37120,
"99.900000" : 52480,
"99.950000" : 64768,
"99.990000" : 75264
}
},
"lat_ns" : {
"min" : 9178,
"max" : 120225,
"mean" : 14536.245858,
"stddev" : 3892.786982,
"N" : 947444
},
"bw_min" : 540352,
"bw_max" : 542264,
"bw_agg" : 100.000000,
"bw_mean" : 541450.857143,
"bw_dev" : 675.183719,
"bw_samples" : 14,
"iops_min" : 135088,
"iops_max" : 135566,
"iops_mean" : 135362.571429,
"iops_stddev" : 168.735439,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 9.314286,
"sys_cpu" : 90.671429,
"ctx" : 7,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.089082,
"20" : 96.324423,
"50" : 3.473665,
"100" : 0.109769,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=86593: Tue Oct 25 23:35:20 2022
read: IOPS=135k, BW=529MiB/s (554MB/s)(3701MiB/7001msec)
slat (nsec): min=1492, max=15409, avg=1579.61, stdev=127.93
clat (nsec): min=270, max=118652, avg=12956.64, stdev=3894.36
lat (usec): min=9, max=120, avg=14.54, stdev= 3.89
clat percentiles (nsec):
| 1.00th=[10048], 5.00th=[10176], 10.00th=[10176], 20.00th=[11968],
| 30.00th=[12096], 40.00th=[12224], 50.00th=[12224], 60.00th=[12352],
| 70.00th=[12352], 80.00th=[14144], 90.00th=[14400], 95.00th=[14656],
| 99.00th=[30592], 99.50th=[37120], 99.90th=[52480], 99.95th=[64768],
| 99.99th=[75264]
bw ( KiB/s): min=540352, max=542264, per=100.00%, avg=541450.86, stdev=675.18, samples=14
iops : min=135088, max=135566, avg=135362.57, stdev=168.74, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.09%, 20=96.32%, 50=3.47%, 100=0.11%
lat (usec) : 250=0.01%
cpu : usr=9.31%, sys=90.67%, ctx=7, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=947443,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=529MiB/s (554MB/s), 529MiB/s-529MiB/s (554MB/s-554MB/s), io=3701MiB (3881MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-xnvme_0.txt"
$ cat /tmp/fio-output_4K_2_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733733,
"timestamp_ms" : 1666733733849,
"time" : "Tue Oct 25 23:35:33 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 2654441472,
"io_kbytes" : 2592228,
"bw_bytes" : 379151760,
"bw" : 370265,
"iops" : 92566.061991,
"runtime" : 7001,
"total_ios" : 648055,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2304,
"max" : 769522,
"mean" : 2713.917901,
"stddev" : 997.462060,
"N" : 648056
},
"clat_ns" : {
"min" : 341,
"max" : 775153,
"mean" : 18398.808790,
"stddev" : 3486.347443,
"N" : 648056,
"percentile" : {
"1.000000" : 16768,
"5.000000" : 16768,
"10.000000" : 17024,
"20.000000" : 17280,
"30.000000" : 17536,
"40.000000" : 17792,
"50.000000" : 17792,
"60.000000" : 18048,
"70.000000" : 18048,
"80.000000" : 18304,
"90.000000" : 18304,
"95.000000" : 20864,
"99.000000" : 34560,
"99.500000" : 43776,
"99.900000" : 50432,
"99.950000" : 63232,
"99.990000" : 75264
}
},
"lat_ns" : {
"min" : 12253,
"max" : 792535,
"mean" : 21112.726690,
"stddev" : 3619.369648,
"N" : 648056
},
"bw_min" : 365080,
"bw_max" : 384096,
"bw_agg" : 100.000000,
"bw_mean" : 370353.714286,
"bw_dev" : 8054.404657,
"bw_samples" : 14,
"iops_min" : 91270,
"iops_max" : 96024,
"iops_mean" : 92588.428571,
"iops_stddev" : 2013.601164,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 11.171429,
"sys_cpu" : 40.400000,
"ctx" : 323954,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 94.840870,
"50" : 5.042628,
"100" : 0.106010,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=86632: Tue Oct 25 23:35:33 2022
read: IOPS=92.6k, BW=362MiB/s (379MB/s)(2531MiB/7001msec)
slat (nsec): min=2304, max=769522, avg=2713.92, stdev=997.46
clat (nsec): min=341, max=775153, avg=18398.81, stdev=3486.35
lat (usec): min=12, max=792, avg=21.11, stdev= 3.62
clat percentiles (nsec):
| 1.00th=[16768], 5.00th=[16768], 10.00th=[17024], 20.00th=[17280],
| 30.00th=[17536], 40.00th=[17792], 50.00th=[17792], 60.00th=[18048],
| 70.00th=[18048], 80.00th=[18304], 90.00th=[18304], 95.00th=[20864],
| 99.00th=[34560], 99.50th=[43776], 99.90th=[50432], 99.95th=[63232],
| 99.99th=[75264]
bw ( KiB/s): min=365080, max=384096, per=100.00%, avg=370353.71, stdev=8054.40, samples=14
iops : min=91270, max=96024, avg=92588.43, stdev=2013.60, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=94.84%, 50=5.04%, 100=0.11%
lat (usec) : 250=0.01%, 1000=0.01%
cpu : usr=11.17%, sys=40.40%, ctx=323954, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=648055,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=362MiB/s (379MB/s), 362MiB/s-362MiB/s (379MB/s-379MB/s), io=2531MiB (2654MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-xnvme_1.txt"
$ cat /tmp/fio-output_4K_2_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733747,
"timestamp_ms" : 1666733747490,
"time" : "Tue Oct 25 23:35:47 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 2740174848,
"io_kbytes" : 2675952,
"bw_bytes" : 391397635,
"bw" : 382224,
"iops" : 95555.920583,
"runtime" : 7001,
"total_ios" : 668987,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2274,
"max" : 30738,
"mean" : 2583.744357,
"stddev" : 358.833559,
"N" : 668987
},
"clat_ns" : {
"min" : 430,
"max" : 156663,
"mean" : 17860.045107,
"stddev" : 3321.104405,
"N" : 668988,
"percentile" : {
"1.000000" : 16768,
"5.000000" : 16768,
"10.000000" : 16768,
"20.000000" : 17024,
"30.000000" : 17024,
"40.000000" : 17024,
"50.000000" : 17280,
"60.000000" : 17280,
"70.000000" : 17536,
"80.000000" : 17536,
"90.000000" : 17792,
"95.000000" : 19584,
"99.000000" : 34048,
"99.500000" : 40704,
"99.900000" : 51456,
"99.950000" : 64768,
"99.990000" : 76288
}
},
"lat_ns" : {
"min" : 12112,
"max" : 159028,
"mean" : 20443.789060,
"stddev" : 3321.208162,
"N" : 668988
},
"bw_min" : 380880,
"bw_max" : 384376,
"bw_agg" : 100.000000,
"bw_mean" : 382313.500000,
"bw_dev" : 1209.400917,
"bw_samples" : 14,
"iops_min" : 95220,
"iops_max" : 96094,
"iops_mean" : 95578.357143,
"iops_stddev" : 302.342548,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.242857,
"sys_cpu" : 41.428571,
"ctx" : 334337,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 95.022474,
"50" : 4.850617,
"100" : 0.116744,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=86671: Tue Oct 25 23:35:47 2022
read: IOPS=95.6k, BW=373MiB/s (391MB/s)(2613MiB/7001msec)
slat (nsec): min=2274, max=30738, avg=2583.74, stdev=358.83
clat (nsec): min=430, max=156663, avg=17860.05, stdev=3321.10
lat (usec): min=12, max=159, avg=20.44, stdev= 3.32
clat percentiles (nsec):
| 1.00th=[16768], 5.00th=[16768], 10.00th=[16768], 20.00th=[17024],
| 30.00th=[17024], 40.00th=[17024], 50.00th=[17280], 60.00th=[17280],
| 70.00th=[17536], 80.00th=[17536], 90.00th=[17792], 95.00th=[19584],
| 99.00th=[34048], 99.50th=[40704], 99.90th=[51456], 99.95th=[64768],
| 99.99th=[76288]
bw ( KiB/s): min=380880, max=384376, per=100.00%, avg=382313.50, stdev=1209.40, samples=14
iops : min=95220, max=96094, avg=95578.36, stdev=302.34, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=95.02%, 50=4.85%, 100=0.12%
lat (usec) : 250=0.01%
cpu : usr=13.24%, sys=41.43%, ctx=334337, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=668987,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=373MiB/s (391MB/s), 373MiB/s-373MiB/s (391MB/s-391MB/s), io=2613MiB (2740MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt': No such file or directory
$ rm /tmp/fio-output_4K_2_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-xnvme_2.txt"
$ cat /tmp/fio-output_4K_2_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733761,
"timestamp_ms" : 1666733761130,
"time" : "Tue Oct 25 23:36:01 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 2729132032,
"io_kbytes" : 2665168,
"bw_bytes" : 389820315,
"bw" : 380683,
"iops" : 95170.689901,
"runtime" : 7001,
"total_ios" : 666290,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2294,
"max" : 39635,
"mean" : 2615.013979,
"stddev" : 376.495980,
"N" : 666291
},
"clat_ns" : {
"min" : 370,
"max" : 191489,
"mean" : 17882.856067,
"stddev" : 3375.558001,
"N" : 666291,
"percentile" : {
"1.000000" : 16768,
"5.000000" : 16768,
"10.000000" : 16768,
"20.000000" : 16768,
"30.000000" : 17024,
"40.000000" : 17024,
"50.000000" : 17280,
"60.000000" : 17536,
"70.000000" : 17536,
"80.000000" : 17536,
"90.000000" : 17792,
"95.000000" : 19584,
"99.000000" : 34048,
"99.500000" : 43264,
"99.900000" : 53504,
"99.950000" : 65280,
"99.990000" : 76288
}
},
"lat_ns" : {
"min" : 11221,
"max" : 193854,
"mean" : 20497.870046,
"stddev" : 3378.217884,
"N" : 666291
},
"bw_min" : 378240,
"bw_max" : 382600,
"bw_agg" : 100.000000,
"bw_mean" : 380772.071429,
"bw_dev" : 1090.263798,
"bw_samples" : 14,
"iops_min" : 94560,
"iops_max" : 95650,
"iops_mean" : 95193.000000,
"iops_stddev" : 272.562484,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.257143,
"sys_cpu" : 39.942857,
"ctx" : 332996,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 95.048703,
"50" : 4.824326,
"100" : 0.118717,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=86711: Tue Oct 25 23:36:01 2022
read: IOPS=95.2k, BW=372MiB/s (390MB/s)(2603MiB/7001msec)
slat (nsec): min=2294, max=39635, avg=2615.01, stdev=376.50
clat (nsec): min=370, max=191489, avg=17882.86, stdev=3375.56
lat (usec): min=11, max=193, avg=20.50, stdev= 3.38
clat percentiles (nsec):
| 1.00th=[16768], 5.00th=[16768], 10.00th=[16768], 20.00th=[16768],
| 30.00th=[17024], 40.00th=[17024], 50.00th=[17280], 60.00th=[17536],
| 70.00th=[17536], 80.00th=[17536], 90.00th=[17792], 95.00th=[19584],
| 99.00th=[34048], 99.50th=[43264], 99.90th=[53504], 99.95th=[65280],
| 99.99th=[76288]
bw ( KiB/s): min=378240, max=382600, per=100.00%, avg=380772.07, stdev=1090.26, samples=14
iops : min=94560, max=95650, avg=95193.00, stdev=272.56, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=95.05%, 50=4.82%, 100=0.12%
lat (usec) : 250=0.01%
cpu : usr=15.26%, sys=39.94%, ctx=332996, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=666290,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=372MiB/s (390MB/s), 372MiB/s-372MiB/s (390MB/s-390MB/s), io=2603MiB (2729MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-reference_0.txt"
$ cat /tmp/fio-output_4K_2_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733774,
"timestamp_ms" : 1666733774795,
"time" : "Tue Oct 25 23:36:14 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 2756984832,
"io_kbytes" : 2692368,
"bw_bytes" : 393798719,
"bw" : 384569,
"iops" : 96142.122554,
"runtime" : 7001,
"total_ios" : 673091,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2394,
"max" : 29245,
"mean" : 2705.547284,
"stddev" : 335.582093,
"N" : 673091
},
"clat_ns" : {
"min" : 802,
"max" : 199354,
"mean" : 17555.780315,
"stddev" : 5106.894427,
"N" : 673092,
"percentile" : {
"1.000000" : 12736,
"5.000000" : 12864,
"10.000000" : 12864,
"20.000000" : 12864,
"30.000000" : 12992,
"40.000000" : 13120,
"50.000000" : 20608,
"60.000000" : 20864,
"70.000000" : 20864,
"80.000000" : 20864,
"90.000000" : 21120,
"95.000000" : 21632,
"99.000000" : 33024,
"99.500000" : 38656,
"99.900000" : 48896,
"99.950000" : 61696,
"99.990000" : 76288
}
},
"lat_ns" : {
"min" : 11712,
"max" : 202269,
"mean" : 20261.327762,
"stddev" : 5287.935425,
"N" : 673092
},
"bw_min" : 383832,
"bw_max" : 385720,
"bw_agg" : 100.000000,
"bw_mean" : 384720.071429,
"bw_dev" : 610.691722,
"bw_samples" : 14,
"iops_min" : 95958,
"iops_max" : 96430,
"iops_mean" : 96179.928571,
"iops_stddev" : 152.704070,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.714286,
"sys_cpu" : 42.514286,
"ctx" : 336367,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 47.780909,
"50" : 52.115241,
"100" : 0.093895,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 963565,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 11496,
"write_ticks" : 0,
"in_queue" : 11496,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=86751: Tue Oct 25 23:36:14 2022
read: IOPS=96.1k, BW=376MiB/s (394MB/s)(2629MiB/7001msec)
slat (nsec): min=2394, max=29245, avg=2705.55, stdev=335.58
clat (nsec): min=802, max=199354, avg=17555.78, stdev=5106.89
lat (usec): min=11, max=202, avg=20.26, stdev= 5.29
clat percentiles (nsec):
| 1.00th=[12736], 5.00th=[12864], 10.00th=[12864], 20.00th=[12864],
| 30.00th=[12992], 40.00th=[13120], 50.00th=[20608], 60.00th=[20864],
| 70.00th=[20864], 80.00th=[20864], 90.00th=[21120], 95.00th=[21632],
| 99.00th=[33024], 99.50th=[38656], 99.90th=[48896], 99.95th=[61696],
| 99.99th=[76288]
bw ( KiB/s): min=383832, max=385720, per=100.00%, avg=384720.07, stdev=610.69, samples=14
iops : min=95958, max=96430, avg=96179.93, stdev=152.70, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=47.78%, 50=52.12%, 100=0.09%
lat (usec) : 250=0.01%
cpu : usr=14.71%, sys=42.51%, ctx=336367, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=673091,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=376MiB/s (394MB/s), 376MiB/s-376MiB/s (394MB/s-394MB/s), io=2629MiB (2757MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=963565/0, merge=0/0, ticks=11496/0, in_queue=11496, util=99.09%
$ rm /tmp/fio-output_4K_2_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-reference_1.txt"
$ cat /tmp/fio-output_4K_2_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733788,
"timestamp_ms" : 1666733788456,
"time" : "Tue Oct 25 23:36:28 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 2764947456,
"io_kbytes" : 2700144,
"bw_bytes" : 394936074,
"bw" : 385679,
"iops" : 96419.654335,
"runtime" : 7001,
"total_ios" : 675034,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2384,
"max" : 46998,
"mean" : 2698.317754,
"stddev" : 340.659642,
"N" : 675035
},
"clat_ns" : {
"min" : 882,
"max" : 136166,
"mean" : 17524.151786,
"stddev" : 5078.250149,
"N" : 675035,
"percentile" : {
"1.000000" : 12736,
"5.000000" : 12864,
"10.000000" : 12864,
"20.000000" : 12992,
"30.000000" : 12992,
"40.000000" : 13120,
"50.000000" : 20608,
"60.000000" : 20608,
"70.000000" : 20864,
"80.000000" : 20864,
"90.000000" : 20864,
"95.000000" : 21632,
"99.000000" : 33024,
"99.500000" : 39168,
"99.900000" : 52480,
"99.950000" : 66048,
"99.990000" : 77312
}
},
"lat_ns" : {
"min" : 12653,
"max" : 139051,
"mean" : 20222.470281,
"stddev" : 5270.579236,
"N" : 675035
},
"bw_min" : 385120,
"bw_max" : 386536,
"bw_agg" : 100.000000,
"bw_mean" : 385827.285714,
"bw_dev" : 381.399332,
"bw_samples" : 14,
"iops_min" : 96280,
"iops_max" : 96634,
"iops_mean" : 96456.714286,
"iops_stddev" : 95.320051,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.000000,
"sys_cpu" : 43.514286,
"ctx" : 337349,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 47.800703,
"50" : 52.086265,
"100" : 0.105328,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 965663,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 11565,
"write_ticks" : 0,
"in_queue" : 11565,
"util" : 99.049128
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=86789: Tue Oct 25 23:36:28 2022
read: IOPS=96.4k, BW=377MiB/s (395MB/s)(2637MiB/7001msec)
slat (nsec): min=2384, max=46998, avg=2698.32, stdev=340.66
clat (nsec): min=882, max=136166, avg=17524.15, stdev=5078.25
lat (usec): min=12, max=139, avg=20.22, stdev= 5.27
clat percentiles (nsec):
| 1.00th=[12736], 5.00th=[12864], 10.00th=[12864], 20.00th=[12992],
| 30.00th=[12992], 40.00th=[13120], 50.00th=[20608], 60.00th=[20608],
| 70.00th=[20864], 80.00th=[20864], 90.00th=[20864], 95.00th=[21632],
| 99.00th=[33024], 99.50th=[39168], 99.90th=[52480], 99.95th=[66048],
| 99.99th=[77312]
bw ( KiB/s): min=385120, max=386536, per=100.00%, avg=385827.29, stdev=381.40, samples=14
iops : min=96280, max=96634, avg=96456.71, stdev=95.32, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=47.80%, 50=52.09%, 100=0.11%
lat (usec) : 250=0.01%
cpu : usr=13.00%, sys=43.51%, ctx=337349, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=675034,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=377MiB/s (395MB/s), 377MiB/s-377MiB/s (395MB/s-395MB/s), io=2637MiB (2765MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=965663/0, merge=0/0, ticks=11565/0, in_queue=11565, util=99.05%
$ rm /tmp/fio-output_4K_2_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt"
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-reference_2.txt"
$ cat /tmp/fio-output_4K_2_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733802,
"timestamp_ms" : 1666733802098,
"time" : "Tue Oct 25 23:36:42 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 2776817664,
"io_kbytes" : 2711736,
"bw_bytes" : 396631576,
"bw" : 387335,
"iops" : 96833.595201,
"runtime" : 7001,
"total_ios" : 677932,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2374,
"max" : 177503,
"mean" : 2666.347233,
"stddev" : 383.992040,
"N" : 677933
},
"clat_ns" : {
"min" : 822,
"max" : 195737,
"mean" : 17461.917285,
"stddev" : 5079.205885,
"N" : 677933,
"percentile" : {
"1.000000" : 12736,
"5.000000" : 12736,
"10.000000" : 12864,
"20.000000" : 12864,
"30.000000" : 12992,
"40.000000" : 13120,
"50.000000" : 20352,
"60.000000" : 20608,
"70.000000" : 20608,
"80.000000" : 20864,
"90.000000" : 20864,
"95.000000" : 21632,
"99.000000" : 33024,
"99.500000" : 38144,
"99.900000" : 50944,
"99.950000" : 65280,
"99.990000" : 77312
}
},
"lat_ns" : {
"min" : 13045,
"max" : 198572,
"mean" : 20128.265079,
"stddev" : 5254.394288,
"N" : 677933
},
"bw_min" : 385704,
"bw_max" : 389835,
"bw_agg" : 100.000000,
"bw_mean" : 387489.428571,
"bw_dev" : 1265.535503,
"bw_samples" : 14,
"iops_min" : 96426,
"iops_max" : 97458,
"iops_mean" : 96872.285714,
"iops_stddev" : 316.254870,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.857143,
"sys_cpu" : 41.557143,
"ctx" : 338781,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 47.810990,
"50" : 52.078527,
"100" : 0.101043,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 970953,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 11596,
"write_ticks" : 0,
"in_queue" : 11596,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=86827: Tue Oct 25 23:36:42 2022
read: IOPS=96.8k, BW=378MiB/s (397MB/s)(2648MiB/7001msec)
slat (nsec): min=2374, max=177503, avg=2666.35, stdev=383.99
clat (nsec): min=822, max=195737, avg=17461.92, stdev=5079.21
lat (usec): min=13, max=198, avg=20.13, stdev= 5.25
clat percentiles (nsec):
| 1.00th=[12736], 5.00th=[12736], 10.00th=[12864], 20.00th=[12864],
| 30.00th=[12992], 40.00th=[13120], 50.00th=[20352], 60.00th=[20608],
| 70.00th=[20608], 80.00th=[20864], 90.00th=[20864], 95.00th=[21632],
| 99.00th=[33024], 99.50th=[38144], 99.90th=[50944], 99.95th=[65280],
| 99.99th=[77312]
bw ( KiB/s): min=385704, max=389835, per=100.00%, avg=387489.43, stdev=1265.54, samples=14
iops : min=96426, max=97458, avg=96872.29, stdev=316.25, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 10=0.01%, 20=47.81%, 50=52.08%, 100=0.10%, 250=0.01%
cpu : usr=14.86%, sys=41.56%, ctx=338781, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=677932,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=378MiB/s (397MB/s), 378MiB/s-378MiB/s (397MB/s-397MB/s), io=2648MiB (2777MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=970953/0, merge=0/0, ticks=11596/0, in_queue=11596, util=99.09%
$ rm /tmp/fio-output_4K_2_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-bdev_aio_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-bdev_aio_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_2_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733817,
"timestamp_ms" : 1666733817120,
"time" : "Tue Oct 25 23:36:57 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3817984000,
"io_kbytes" : 3728500,
"bw_bytes" : 545348378,
"bw" : 532566,
"iops" : 133141.551207,
"runtime" : 7001,
"total_ios" : 932124,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1503,
"max" : 15279,
"mean" : 1580.962068,
"stddev" : 125.314578,
"N" : 932125
},
"clat_ns" : {
"min" : 261,
"max" : 116468,
"mean" : 12250.310121,
"stddev" : 3714.120484,
"N" : 932125,
"percentile" : {
"1.000000" : 10560,
"5.000000" : 10560,
"10.000000" : 10560,
"20.000000" : 10688,
"30.000000" : 10688,
"40.000000" : 10816,
"50.000000" : 12224,
"60.000000" : 12352,
"70.000000" : 12352,
"80.000000" : 12480,
"90.000000" : 12608,
"95.000000" : 13120,
"99.000000" : 29568,
"99.500000" : 37120,
"99.900000" : 45312,
"99.950000" : 63232,
"99.990000" : 74240
}
},
"lat_ns" : {
"min" : 9247,
"max" : 118081,
"mean" : 13831.272190,
"stddev" : 3720.542513,
"N" : 932125
},
"bw_min" : 531104,
"bw_max" : 534640,
"bw_agg" : 100.000000,
"bw_mean" : 532689.428571,
"bw_dev" : 1101.338606,
"bw_samples" : 14,
"iops_min" : 132776,
"iops_max" : 133660,
"iops_mean" : 133172.214286,
"iops_stddev" : 275.363306,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 84.400000,
"sys_cpu" : 15.585714,
"ctx" : 4,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 49.981869,
"2" : 50.018131,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.049886,
"20" : 96.550674,
"50" : 3.300312,
"100" : 0.096232,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=86875: Tue Oct 25 23:36:57 2022
read: IOPS=133k, BW=520MiB/s (545MB/s)(3641MiB/7001msec)
slat (nsec): min=1503, max=15279, avg=1580.96, stdev=125.31
clat (nsec): min=261, max=116468, avg=12250.31, stdev=3714.12
lat (usec): min=9, max=118, avg=13.83, stdev= 3.72
clat percentiles (nsec):
| 1.00th=[10560], 5.00th=[10560], 10.00th=[10560], 20.00th=[10688],
| 30.00th=[10688], 40.00th=[10816], 50.00th=[12224], 60.00th=[12352],
| 70.00th=[12352], 80.00th=[12480], 90.00th=[12608], 95.00th=[13120],
| 99.00th=[29568], 99.50th=[37120], 99.90th=[45312], 99.95th=[63232],
| 99.99th=[74240]
bw ( KiB/s): min=531104, max=534640, per=100.00%, avg=532689.43, stdev=1101.34, samples=14
iops : min=132776, max=133660, avg=133172.21, stdev=275.36, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.05%, 20=96.55%, 50=3.30%, 100=0.10%
lat (usec) : 250=0.01%
cpu : usr=84.40%, sys=15.59%, ctx=4, majf=0, minf=0
IO depths : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=932124,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=520MiB/s (545MB/s), 520MiB/s-520MiB/s (545MB/s-545MB/s), io=3641MiB (3818MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-bdev_aio_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-bdev_aio_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_2_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733832,
"timestamp_ms" : 1666733832158,
"time" : "Tue Oct 25 23:37:12 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3812896768,
"io_kbytes" : 3723532,
"bw_bytes" : 544621735,
"bw" : 531857,
"iops" : 132964.147979,
"runtime" : 7001,
"total_ios" : 930882,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1542,
"max" : 43872,
"mean" : 1620.137457,
"stddev" : 131.643678,
"N" : 930882
},
"clat_ns" : {
"min" : 240,
"max" : 1034130,
"mean" : 12198.356105,
"stddev" : 4029.463553,
"N" : 930882,
"percentile" : {
"1.000000" : 10432,
"5.000000" : 10432,
"10.000000" : 10560,
"20.000000" : 10560,
"30.000000" : 10688,
"40.000000" : 10688,
"50.000000" : 12224,
"60.000000" : 12352,
"70.000000" : 12352,
"80.000000" : 12480,
"90.000000" : 12480,
"95.000000" : 13120,
"99.000000" : 29568,
"99.500000" : 37120,
"99.900000" : 50432,
"99.950000" : 63744,
"99.990000" : 74240
}
},
"lat_ns" : {
"min" : 9217,
"max" : 1035763,
"mean" : 13818.493562,
"stddev" : 4036.311352,
"N" : 930882
},
"bw_min" : 528936,
"bw_max" : 533304,
"bw_agg" : 100.000000,
"bw_mean" : 531980.714286,
"bw_dev" : 1458.161879,
"bw_samples" : 14,
"iops_min" : 132234,
"iops_max" : 133326,
"iops_mean" : 132995.142857,
"iops_stddev" : 364.505747,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 86.657143,
"sys_cpu" : 13.300000,
"ctx" : 41,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 49.984746,
"2" : 50.015254,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.049738,
"20" : 96.555847,
"50" : 3.289031,
"100" : 0.102483,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=86924: Tue Oct 25 23:37:12 2022
read: IOPS=133k, BW=519MiB/s (545MB/s)(3636MiB/7001msec)
slat (nsec): min=1542, max=43872, avg=1620.14, stdev=131.64
clat (nsec): min=240, max=1034.1k, avg=12198.36, stdev=4029.46
lat (usec): min=9, max=1035, avg=13.82, stdev= 4.04
clat percentiles (nsec):
| 1.00th=[10432], 5.00th=[10432], 10.00th=[10560], 20.00th=[10560],
| 30.00th=[10688], 40.00th=[10688], 50.00th=[12224], 60.00th=[12352],
| 70.00th=[12352], 80.00th=[12480], 90.00th=[12480], 95.00th=[13120],
| 99.00th=[29568], 99.50th=[37120], 99.90th=[50432], 99.95th=[63744],
| 99.99th=[74240]
bw ( KiB/s): min=528936, max=533304, per=100.00%, avg=531980.71, stdev=1458.16, samples=14
iops : min=132234, max=133326, avg=132995.14, stdev=364.51, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=0.05%, 20=96.56%, 50=3.29%, 100=0.10%
lat (usec) : 250=0.01%
lat (msec) : 2=0.01%
cpu : usr=86.66%, sys=13.30%, ctx=41, majf=0, minf=0
IO depths : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=930882,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=519MiB/s (545MB/s), 519MiB/s-519MiB/s (545MB/s-545MB/s), io=3636MiB (3813MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-bdev_aio_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-bdev_aio_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733070,
"timestamp_ms" : 1666733070951,
"time" : "Tue Oct 25 23:24:30 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 1847742464,
"io_kbytes" : 1804436,
"bw_bytes" : 263925505,
"bw" : 257739,
"iops" : 64434.937866,
"runtime" : 7001,
"total_ios" : 451109,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2314,
"max" : 18234,
"mean" : 2506.600283,
"stddev" : 209.404360,
"N" : 451109
},
"clat_ns" : {
"min" : 341,
"max" : 183043,
"mean" : 12344.901031,
"stddev" : 2052.638325,
"N" : 451109,
"percentile" : {
"1.000000" : 11840,
"5.000000" : 11840,
"10.000000" : 11840,
"20.000000" : 11968,
"30.000000" : 11968,
"40.000000" : 11968,
"50.000000" : 12096,
"60.000000" : 12096,
"70.000000" : 12224,
"80.000000" : 12224,
"90.000000" : 12352,
"95.000000" : 12480,
"99.000000" : 22144,
"99.500000" : 24448,
"99.900000" : 31616,
"99.950000" : 49920,
"99.990000" : 64256
}
},
"lat_ns" : {
"min" : 11482,
"max" : 185648,
"mean" : 14851.501313,
"stddev" : 2066.668313,
"N" : 451109
},
"bw_min" : 257312,
"bw_max" : 258252,
"bw_agg" : 100.000000,
"bw_mean" : 257799.142857,
"bw_dev" : 320.139118,
"bw_samples" : 14,
"iops_min" : 64328,
"iops_max" : 64563,
"iops_mean" : 64449.785714,
"iops_stddev" : 80.034780,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 10.671429,
"sys_cpu" : 36.157143,
"ctx" : 451088,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 98.461569,
"50" : 1.477913,
"100" : 0.051650,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=84584: Tue Oct 25 23:24:30 2022
read: IOPS=64.4k, BW=252MiB/s (264MB/s)(1762MiB/7001msec)
slat (nsec): min=2314, max=18234, avg=2506.60, stdev=209.40
clat (nsec): min=341, max=183043, avg=12344.90, stdev=2052.64
lat (usec): min=11, max=185, avg=14.85, stdev= 2.07
clat percentiles (nsec):
| 1.00th=[11840], 5.00th=[11840], 10.00th=[11840], 20.00th=[11968],
| 30.00th=[11968], 40.00th=[11968], 50.00th=[12096], 60.00th=[12096],
| 70.00th=[12224], 80.00th=[12224], 90.00th=[12352], 95.00th=[12480],
| 99.00th=[22144], 99.50th=[24448], 99.90th=[31616], 99.95th=[49920],
| 99.99th=[64256]
bw ( KiB/s): min=257312, max=258252, per=100.00%, avg=257799.14, stdev=320.14, samples=14
iops : min=64328, max=64563, avg=64449.79, stdev=80.03, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=98.46%, 50=1.48%, 100=0.05%
lat (usec) : 250=0.01%
cpu : usr=10.67%, sys=36.16%, ctx=451088, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=451109,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=252MiB/s (264MB/s), 252MiB/s-252MiB/s (264MB/s-264MB/s), io=1762MiB (1848MB), run=7001-7001msec
$ cat /tmp/fio-output_4K_2_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733847,
"timestamp_ms" : 1666733847292,
"time" : "Tue Oct 25 23:37:27 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3812159488,
"io_kbytes" : 3722812,
"bw_bytes" : 544516424,
"bw" : 531754,
"iops" : 132938.437366,
"runtime" : 7001,
"total_ios" : 930702,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1542,
"max" : 16170,
"mean" : 1619.645983,
"stddev" : 121.460440,
"N" : 930702
},
"clat_ns" : {
"min" : 251,
"max" : 176772,
"mean" : 12194.889915,
"stddev" : 3779.824468,
"N" : 930702,
"percentile" : {
"1.000000" : 10432,
"5.000000" : 10560,
"10.000000" : 10560,
"20.000000" : 10560,
"30.000000" : 10688,
"40.000000" : 10688,
"50.000000" : 12224,
"60.000000" : 12352,
"70.000000" : 12352,
"80.000000" : 12480,
"90.000000" : 12480,
"95.000000" : 12992,
"99.000000" : 29568,
"99.500000" : 37120,
"99.900000" : 50432,
"99.950000" : 63744,
"99.990000" : 74240
}
},
"lat_ns" : {
"min" : 9257,
"max" : 178395,
"mean" : 13814.535899,
"stddev" : 3787.101803,
"N" : 930702
},
"bw_min" : 530488,
"bw_max" : 532609,
"bw_agg" : 100.000000,
"bw_mean" : 531886.928571,
"bw_dev" : 581.030446,
"bw_samples" : 14,
"iops_min" : 132622,
"iops_max" : 133152,
"iops_mean" : 132971.714286,
"iops_stddev" : 145.339617,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 84.900000,
"sys_cpu" : 15.071429,
"ctx" : 16,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 49.984635,
"2" : 50.015365,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.048458,
"20" : 96.556255,
"50" : 3.289452,
"100" : 0.102288,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=86975: Tue Oct 25 23:37:27 2022
read: IOPS=133k, BW=519MiB/s (545MB/s)(3636MiB/7001msec)
slat (nsec): min=1542, max=16170, avg=1619.65, stdev=121.46
clat (nsec): min=251, max=176772, avg=12194.89, stdev=3779.82
lat (usec): min=9, max=178, avg=13.81, stdev= 3.79
clat percentiles (nsec):
| 1.00th=[10432], 5.00th=[10560], 10.00th=[10560], 20.00th=[10560],
| 30.00th=[10688], 40.00th=[10688], 50.00th=[12224], 60.00th=[12352],
| 70.00th=[12352], 80.00th=[12480], 90.00th=[12480], 95.00th=[12992],
| 99.00th=[29568], 99.50th=[37120], 99.90th=[50432], 99.95th=[63744],
| 99.99th=[74240]
bw ( KiB/s): min=530488, max=532609, per=100.00%, avg=531886.93, stdev=581.03, samples=14
iops : min=132622, max=133152, avg=132971.71, stdev=145.34, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.05%, 20=96.56%, 50=3.29%, 100=0.10%
lat (usec) : 250=0.01%
cpu : usr=84.90%, sys=15.07%, ctx=16, majf=0, minf=0
IO depths : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=930702,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=519MiB/s (545MB/s), 519MiB/s-519MiB/s (545MB/s-545MB/s), io=3636MiB (3812MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733862,
"timestamp_ms" : 1666733862396,
"time" : "Tue Oct 25 23:37:42 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3758456832,
"io_kbytes" : 3670368,
"bw_bytes" : 536845712,
"bw" : 524263,
"iops" : 131065.704899,
"runtime" : 7001,
"total_ios" : 917591,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1542,
"max" : 1290761,
"mean" : 1616.681889,
"stddev" : 1355.420376,
"N" : 917592
},
"clat_ns" : {
"min" : 691,
"max" : 1291843,
"mean" : 12424.666600,
"stddev" : 3999.828740,
"N" : 917592,
"percentile" : {
"1.000000" : 10560,
"5.000000" : 10560,
"10.000000" : 10688,
"20.000000" : 10816,
"30.000000" : 10944,
"40.000000" : 10944,
"50.000000" : 12352,
"60.000000" : 12480,
"70.000000" : 12608,
"80.000000" : 12736,
"90.000000" : 12736,
"95.000000" : 13376,
"99.000000" : 29824,
"99.500000" : 37120,
"99.900000" : 50944,
"99.950000" : 63744,
"99.990000" : 75264
}
},
"lat_ns" : {
"min" : 9447,
"max" : 1294137,
"mean" : 14041.348489,
"stddev" : 4225.469205,
"N" : 917592
},
"bw_min" : 523344,
"bw_max" : 524897,
"bw_agg" : 100.000000,
"bw_mean" : 524383.500000,
"bw_dev" : 486.058995,
"bw_samples" : 14,
"iops_min" : 130836,
"iops_max" : 131224,
"iops_mean" : 131095.857143,
"iops_stddev" : 121.494449,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 32.271429,
"sys_cpu" : 67.685714,
"ctx" : 43,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 49.985451,
"2" : 50.014549,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.042394,
"20" : 96.514896,
"50" : 3.329261,
"100" : 0.110616,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87024: Tue Oct 25 23:37:42 2022
read: IOPS=131k, BW=512MiB/s (537MB/s)(3584MiB/7001msec)
slat (nsec): min=1542, max=1290.8k, avg=1616.68, stdev=1355.42
clat (nsec): min=691, max=1291.8k, avg=12424.67, stdev=3999.83
lat (usec): min=9, max=1294, avg=14.04, stdev= 4.23
clat percentiles (nsec):
| 1.00th=[10560], 5.00th=[10560], 10.00th=[10688], 20.00th=[10816],
| 30.00th=[10944], 40.00th=[10944], 50.00th=[12352], 60.00th=[12480],
| 70.00th=[12608], 80.00th=[12736], 90.00th=[12736], 95.00th=[13376],
| 99.00th=[29824], 99.50th=[37120], 99.90th=[50944], 99.95th=[63744],
| 99.99th=[75264]
bw ( KiB/s): min=523344, max=524897, per=100.00%, avg=524383.50, stdev=486.06, samples=14
iops : min=130836, max=131224, avg=131095.86, stdev=121.49, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 4=0.01%, 10=0.04%, 20=96.51%, 50=3.33%, 100=0.11%
lat (usec) : 250=0.01%
lat (msec) : 2=0.01%
cpu : usr=32.27%, sys=67.69%, ctx=43, majf=0, minf=0
IO depths : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=917591,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=512MiB/s (537MB/s), 512MiB/s-512MiB/s (537MB/s-537MB/s), io=3584MiB (3758MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733877,
"timestamp_ms" : 1666733877524,
"time" : "Tue Oct 25 23:37:57 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3756961792,
"io_kbytes" : 3668908,
"bw_bytes" : 536632165,
"bw" : 524054,
"iops" : 131013.569490,
"runtime" : 7001,
"total_ios" : 917226,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1543,
"max" : 16892,
"mean" : 1635.534015,
"stddev" : 122.054841,
"N" : 917227
},
"clat_ns" : {
"min" : 711,
"max" : 1299107,
"mean" : 12389.945191,
"stddev" : 4132.412283,
"N" : 917227,
"percentile" : {
"1.000000" : 10560,
"5.000000" : 10688,
"10.000000" : 10688,
"20.000000" : 10688,
"30.000000" : 10816,
"40.000000" : 11072,
"50.000000" : 12352,
"60.000000" : 12480,
"70.000000" : 12480,
"80.000000" : 12608,
"90.000000" : 12864,
"95.000000" : 13376,
"99.000000" : 29568,
"99.500000" : 36608,
"99.900000" : 42240,
"99.950000" : 62208,
"99.990000" : 75264
}
},
"lat_ns" : {
"min" : 9368,
"max" : 1300770,
"mean" : 14025.479206,
"stddev" : 4142.473008,
"N" : 917227
},
"bw_min" : 523160,
"bw_max" : 525571,
"bw_agg" : 100.000000,
"bw_mean" : 524175.642857,
"bw_dev" : 851.873917,
"bw_samples" : 14,
"iops_min" : 130790,
"iops_max" : 131392,
"iops_mean" : 131043.857143,
"iops_stddev" : 212.874053,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 35.357143,
"sys_cpu" : 64.614286,
"ctx" : 40,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 49.978522,
"2" : 50.021478,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.056911,
"20" : 96.554502,
"50" : 3.304638,
"100" : 0.081332,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87073: Tue Oct 25 23:37:57 2022
read: IOPS=131k, BW=512MiB/s (537MB/s)(3583MiB/7001msec)
slat (nsec): min=1543, max=16892, avg=1635.53, stdev=122.05
clat (nsec): min=711, max=1299.1k, avg=12389.95, stdev=4132.41
lat (usec): min=9, max=1300, avg=14.03, stdev= 4.14
clat percentiles (nsec):
| 1.00th=[10560], 5.00th=[10688], 10.00th=[10688], 20.00th=[10688],
| 30.00th=[10816], 40.00th=[11072], 50.00th=[12352], 60.00th=[12480],
| 70.00th=[12480], 80.00th=[12608], 90.00th=[12864], 95.00th=[13376],
| 99.00th=[29568], 99.50th=[36608], 99.90th=[42240], 99.95th=[62208],
| 99.99th=[75264]
bw ( KiB/s): min=523160, max=525571, per=100.00%, avg=524175.64, stdev=851.87, samples=14
iops : min=130790, max=131392, avg=131043.86, stdev=212.87, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 4=0.01%, 10=0.06%, 20=96.55%, 50=3.30%, 100=0.08%
lat (usec) : 250=0.01%
lat (msec) : 2=0.01%
cpu : usr=35.36%, sys=64.61%, ctx=40, majf=0, minf=0
IO depths : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=917226,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=512MiB/s (537MB/s), 512MiB/s-512MiB/s (537MB/s-537MB/s), io=3583MiB (3757MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733892,
"timestamp_ms" : 1666733892681,
"time" : "Tue Oct 25 23:38:12 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 3749478400,
"io_kbytes" : 3661600,
"bw_bytes" : 535563262,
"bw" : 523010,
"iops" : 130752.606770,
"runtime" : 7001,
"total_ios" : 915399,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1552,
"max" : 175259,
"mean" : 1626.508294,
"stddev" : 223.366531,
"N" : 915399
},
"clat_ns" : {
"min" : 711,
"max" : 181591,
"mean" : 12431.741996,
"stddev" : 3780.590270,
"N" : 915399,
"percentile" : {
"1.000000" : 10560,
"5.000000" : 10560,
"10.000000" : 10688,
"20.000000" : 10816,
"30.000000" : 10944,
"40.000000" : 11072,
"50.000000" : 12352,
"60.000000" : 12480,
"70.000000" : 12608,
"80.000000" : 12736,
"90.000000" : 12864,
"95.000000" : 13376,
"99.000000" : 29824,
"99.500000" : 37120,
"99.900000" : 50432,
"99.950000" : 63744,
"99.990000" : 80384
}
},
"lat_ns" : {
"min" : 9377,
"max" : 188053,
"mean" : 14058.250290,
"stddev" : 3793.051475,
"N" : 915399
},
"bw_min" : 521616,
"bw_max" : 524296,
"bw_agg" : 100.000000,
"bw_mean" : 523131.285714,
"bw_dev" : 699.172965,
"bw_samples" : 14,
"iops_min" : 130404,
"iops_max" : 131074,
"iops_mean" : 130782.785714,
"iops_stddev" : 174.475116,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 34.957143,
"sys_cpu" : 65.014286,
"ctx" : 17,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 49.974055,
"2" : 50.025945,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.063470,
"20" : 96.510811,
"50" : 3.320410,
"100" : 0.101267,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87122: Tue Oct 25 23:38:12 2022
read: IOPS=131k, BW=511MiB/s (536MB/s)(3576MiB/7001msec)
slat (nsec): min=1552, max=175259, avg=1626.51, stdev=223.37
clat (nsec): min=711, max=181591, avg=12431.74, stdev=3780.59
lat (usec): min=9, max=188, avg=14.06, stdev= 3.79
clat percentiles (nsec):
| 1.00th=[10560], 5.00th=[10560], 10.00th=[10688], 20.00th=[10816],
| 30.00th=[10944], 40.00th=[11072], 50.00th=[12352], 60.00th=[12480],
| 70.00th=[12608], 80.00th=[12736], 90.00th=[12864], 95.00th=[13376],
| 99.00th=[29824], 99.50th=[37120], 99.90th=[50432], 99.95th=[63744],
| 99.99th=[80384]
bw ( KiB/s): min=521616, max=524296, per=100.00%, avg=523131.29, stdev=699.17, samples=14
iops : min=130404, max=131074, avg=130782.79, stdev=174.48, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 4=0.01%, 10=0.06%, 20=96.51%, 50=3.32%, 100=0.10%
lat (usec) : 250=0.01%
cpu : usr=34.96%, sys=65.01%, ctx=17, majf=0, minf=0
IO depths : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=915399,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=511MiB/s (536MB/s), 511MiB/s-511MiB/s (536MB/s-536MB/s), io=3576MiB (3749MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory
$ rm /tmp/fio-output_4K_2_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-xnvme_0.txt"
$ cat /tmp/fio-output_4K_2_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733906,
"timestamp_ms" : 1666733906474,
"time" : "Tue Oct 25 23:38:26 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 2690433024,
"io_kbytes" : 2627376,
"bw_bytes" : 384292675,
"bw" : 375285,
"iops" : 93821.311241,
"runtime" : 7001,
"total_ios" : 656843,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2504,
"max" : 178896,
"mean" : 2877.458288,
"stddev" : 468.361840,
"N" : 656843
},
"clat_ns" : {
"min" : 951,
"max" : 1352827,
"mean" : 17919.024456,
"stddev" : 5851.439789,
"N" : 656844,
"percentile" : {
"1.000000" : 12736,
"5.000000" : 12864,
"10.000000" : 12992,
"20.000000" : 12992,
"30.000000" : 13120,
"40.000000" : 13120,
"50.000000" : 21376,
"60.000000" : 21376,
"70.000000" : 21376,
"80.000000" : 21632,
"90.000000" : 21632,
"95.000000" : 22144,
"99.000000" : 35072,
"99.500000" : 40192,
"99.900000" : 50944,
"99.950000" : 64256,
"99.990000" : 77312
}
},
"lat_ns" : {
"min" : 12153,
"max" : 1355933,
"mean" : 20796.483029,
"stddev" : 6068.884642,
"N" : 656844
},
"bw_min" : 374056,
"bw_max" : 377312,
"bw_agg" : 100.000000,
"bw_mean" : 375377.071429,
"bw_dev" : 779.433371,
"bw_samples" : 14,
"iops_min" : 93514,
"iops_max" : 94328,
"iops_mean" : 93844.214286,
"iops_stddev" : 194.825436,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.200000,
"sys_cpu" : 45.857143,
"ctx" : 328203,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 47.701962,
"50" : 52.184160,
"100" : 0.103678,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=87161: Tue Oct 25 23:38:26 2022
read: IOPS=93.8k, BW=366MiB/s (384MB/s)(2566MiB/7001msec)
slat (nsec): min=2504, max=178896, avg=2877.46, stdev=468.36
clat (nsec): min=951, max=1352.8k, avg=17919.02, stdev=5851.44
lat (usec): min=12, max=1355, avg=20.80, stdev= 6.07
clat percentiles (nsec):
| 1.00th=[12736], 5.00th=[12864], 10.00th=[12992], 20.00th=[12992],
| 30.00th=[13120], 40.00th=[13120], 50.00th=[21376], 60.00th=[21376],
| 70.00th=[21376], 80.00th=[21632], 90.00th=[21632], 95.00th=[22144],
| 99.00th=[35072], 99.50th=[40192], 99.90th=[50944], 99.95th=[64256],
| 99.99th=[77312]
bw ( KiB/s): min=374056, max=377312, per=100.00%, avg=375377.07, stdev=779.43, samples=14
iops : min=93514, max=94328, avg=93844.21, stdev=194.83, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=47.70%, 50=52.18%, 100=0.10%
lat (usec) : 250=0.01%
lat (msec) : 2=0.01%
cpu : usr=15.20%, sys=45.86%, ctx=328203, majf=0, minf=1
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=656843,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=366MiB/s (384MB/s), 366MiB/s-366MiB/s (384MB/s-384MB/s), io=2566MiB (2690MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-xnvme_1.txt"
$ cat /tmp/fio-output_4K_2_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733920,
"timestamp_ms" : 1666733920115,
"time" : "Tue Oct 25 23:38:40 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 2698158080,
"io_kbytes" : 2634920,
"bw_bytes" : 385396097,
"bw" : 376363,
"iops" : 94090.701328,
"runtime" : 7001,
"total_ios" : 658729,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2534,
"max" : 47179,
"mean" : 2900.782150,
"stddev" : 418.117679,
"N" : 658729
},
"clat_ns" : {
"min" : 921,
"max" : 118352,
"mean" : 17829.021652,
"stddev" : 5403.400026,
"N" : 658730,
"percentile" : {
"1.000000" : 12608,
"5.000000" : 12736,
"10.000000" : 12736,
"20.000000" : 12864,
"30.000000" : 12864,
"40.000000" : 12992,
"50.000000" : 21376,
"60.000000" : 21376,
"70.000000" : 21376,
"80.000000" : 21632,
"90.000000" : 21632,
"95.000000" : 22144,
"99.000000" : 34560,
"99.500000" : 39680,
"99.900000" : 48896,
"99.950000" : 61184,
"99.990000" : 76288
}
},
"lat_ns" : {
"min" : 12253,
"max" : 121407,
"mean" : 20729.804006,
"stddev" : 5628.504721,
"N" : 658730
},
"bw_min" : 375336,
"bw_max" : 377432,
"bw_agg" : 100.000000,
"bw_mean" : 376463.000000,
"bw_dev" : 622.233570,
"bw_samples" : 14,
"iops_min" : 93834,
"iops_max" : 94358,
"iops_mean" : 94115.714286,
"iops_stddev" : 155.503860,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.242857,
"sys_cpu" : 46.571429,
"ctx" : 329136,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.010019,
"20" : 47.720686,
"50" : 52.176995,
"100" : 0.090629,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=87201: Tue Oct 25 23:38:40 2022
read: IOPS=94.1k, BW=368MiB/s (385MB/s)(2573MiB/7001msec)
slat (nsec): min=2534, max=47179, avg=2900.78, stdev=418.12
clat (nsec): min=921, max=118352, avg=17829.02, stdev=5403.40
lat (usec): min=12, max=121, avg=20.73, stdev= 5.63
clat percentiles (nsec):
| 1.00th=[12608], 5.00th=[12736], 10.00th=[12736], 20.00th=[12864],
| 30.00th=[12864], 40.00th=[12992], 50.00th=[21376], 60.00th=[21376],
| 70.00th=[21376], 80.00th=[21632], 90.00th=[21632], 95.00th=[22144],
| 99.00th=[34560], 99.50th=[39680], 99.90th=[48896], 99.95th=[61184],
| 99.99th=[76288]
bw ( KiB/s): min=375336, max=377432, per=100.00%, avg=376463.00, stdev=622.23, samples=14
iops : min=93834, max=94358, avg=94115.71, stdev=155.50, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=47.72%, 50=52.18%
lat (usec) : 100=0.09%, 250=0.01%
cpu : usr=15.24%, sys=46.57%, ctx=329136, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=658729,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=368MiB/s (385MB/s), 368MiB/s-368MiB/s (385MB/s-385MB/s), io=2573MiB (2698MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_2_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-xnvme_2.txt"
$ cat /tmp/fio-output_4K_2_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733933,
"timestamp_ms" : 1666733933796,
"time" : "Tue Oct 25 23:38:53 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 2710253568,
"io_kbytes" : 2646732,
"bw_bytes" : 387123777,
"bw" : 378050,
"iops" : 94512.498215,
"runtime" : 7001,
"total_ios" : 661682,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2494,
"max" : 179777,
"mean" : 2858.011643,
"stddev" : 548.309512,
"N" : 661682
},
"clat_ns" : {
"min" : 902,
"max" : 195327,
"mean" : 17806.183939,
"stddev" : 5345.044073,
"N" : 661683,
"percentile" : {
"1.000000" : 12736,
"5.000000" : 12864,
"10.000000" : 12864,
"20.000000" : 12992,
"30.000000" : 12992,
"40.000000" : 13120,
"50.000000" : 21120,
"60.000000" : 21376,
"70.000000" : 21376,
"80.000000" : 21376,
"90.000000" : 21376,
"95.000000" : 21888,
"99.000000" : 35072,
"99.500000" : 40192,
"99.900000" : 50944,
"99.950000" : 64768,
"99.990000" : 77312
}
},
"lat_ns" : {
"min" : 12413,
"max" : 201298,
"mean" : 20664.195956,
"stddev" : 5573.558208,
"N" : 661683
},
"bw_min" : 377240,
"bw_max" : 379552,
"bw_agg" : 100.000000,
"bw_mean" : 378145.928571,
"bw_dev" : 652.061638,
"bw_samples" : 14,
"iops_min" : 94310,
"iops_max" : 94888,
"iops_mean" : 94536.428571,
"iops_stddev" : 163.055778,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.442857,
"sys_cpu" : 43.571429,
"ctx" : 330603,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 47.726551,
"50" : 52.157683,
"100" : 0.103373,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=87241: Tue Oct 25 23:38:53 2022
read: IOPS=94.5k, BW=369MiB/s (387MB/s)(2585MiB/7001msec)
slat (nsec): min=2494, max=179777, avg=2858.01, stdev=548.31
clat (nsec): min=902, max=195327, avg=17806.18, stdev=5345.04
lat (usec): min=12, max=201, avg=20.66, stdev= 5.57
clat percentiles (nsec):
| 1.00th=[12736], 5.00th=[12864], 10.00th=[12864], 20.00th=[12992],
| 30.00th=[12992], 40.00th=[13120], 50.00th=[21120], 60.00th=[21376],
| 70.00th=[21376], 80.00th=[21376], 90.00th=[21376], 95.00th=[21888],
| 99.00th=[35072], 99.50th=[40192], 99.90th=[50944], 99.95th=[64768],
| 99.99th=[77312]
bw ( KiB/s): min=377240, max=379552, per=100.00%, avg=378145.93, stdev=652.06, samples=14
iops : min=94310, max=94888, avg=94536.43, stdev=163.06, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=47.73%, 50=52.16%, 100=0.10%
lat (usec) : 250=0.01%
cpu : usr=17.44%, sys=43.57%, ctx=330603, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=661682,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=369MiB/s (387MB/s), 369MiB/s-369MiB/s (387MB/s-387MB/s), io=2585MiB (2710MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt"
$ cat /tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733947,
"timestamp_ms" : 1666733947431,
"time" : "Tue Oct 25 23:39:07 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 4239355904,
"io_kbytes" : 4139996,
"bw_bytes" : 605535766,
"bw" : 591343,
"iops" : 147835.309242,
"runtime" : 7001,
"total_ios" : 1034995,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1863,
"max" : 68268,
"mean" : 2134.228236,
"stddev" : 423.418030,
"N" : 1034996
},
"clat_ns" : {
"min" : 210,
"max" : 170530,
"mean" : 24422.255110,
"stddev" : 7557.409280,
"N" : 1034998,
"percentile" : {
"1.000000" : 14528,
"5.000000" : 17024,
"10.000000" : 17024,
"20.000000" : 17792,
"30.000000" : 18048,
"40.000000" : 18560,
"50.000000" : 28032,
"60.000000" : 28032,
"70.000000" : 28288,
"80.000000" : 28800,
"90.000000" : 29056,
"95.000000" : 34048,
"99.000000" : 51968,
"99.500000" : 53504,
"99.900000" : 82432,
"99.950000" : 97792,
"99.990000" : 112128
}
},
"lat_ns" : {
"min" : 13415,
"max" : 172494,
"mean" : 26556.483509,
"stddev" : 7660.333802,
"N" : 1034998
},
"bw_min" : 590184,
"bw_max" : 592416,
"bw_agg" : 100.000000,
"bw_mean" : 591485.071429,
"bw_dev" : 642.692948,
"bw_samples" : 14,
"iops_min" : 147546,
"iops_max" : 148104,
"iops_mean" : 147871.214286,
"iops_stddev" : 160.657389,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.742857,
"sys_cpu" : 57.785714,
"ctx" : 266084,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 44.522920,
"50" : 53.856106,
"100" : 1.573244,
"250" : 0.045604,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=87281: Tue Oct 25 23:39:07 2022
read: IOPS=148k, BW=577MiB/s (606MB/s)(4043MiB/7001msec)
slat (nsec): min=1863, max=68268, avg=2134.23, stdev=423.42
clat (nsec): min=210, max=170530, avg=24422.26, stdev=7557.41
lat (usec): min=13, max=172, avg=26.56, stdev= 7.66
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18],
| 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 29], 60.00th=[ 29],
| 70.00th=[ 29], 80.00th=[ 29], 90.00th=[ 30], 95.00th=[ 35],
| 99.00th=[ 52], 99.50th=[ 54], 99.90th=[ 83], 99.95th=[ 98],
| 99.99th=[ 113]
bw ( KiB/s): min=590184, max=592416, per=100.00%, avg=591485.07, stdev=642.69, samples=14
iops : min=147546, max=148104, avg=147871.21, stdev=160.66, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=44.52%, 50=53.86%, 100=1.57%
lat (usec) : 250=0.05%
cpu : usr=14.74%, sys=57.79%, ctx=266084, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1034995,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=577MiB/s (606MB/s), 577MiB/s-577MiB/s (606MB/s-606MB/s), io=4043MiB (4239MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt"
$ cat /tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733961,
"timestamp_ms" : 1666733961073,
"time" : "Tue Oct 25 23:39:21 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 4248465408,
"io_kbytes" : 4148892,
"bw_bytes" : 606836938,
"bw" : 592614,
"iops" : 148152.978146,
"runtime" : 7001,
"total_ios" : 1037219,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1874,
"max" : 27602,
"mean" : 2143.573774,
"stddev" : 413.583393,
"N" : 1037220
},
"clat_ns" : {
"min" : 2985,
"max" : 204163,
"mean" : 24353.154686,
"stddev" : 7561.186398,
"N" : 1037222,
"percentile" : {
"1.000000" : 16768,
"5.000000" : 17024,
"10.000000" : 17024,
"20.000000" : 17792,
"30.000000" : 17792,
"40.000000" : 18560,
"50.000000" : 28032,
"60.000000" : 28032,
"70.000000" : 28288,
"80.000000" : 28800,
"90.000000" : 29056,
"95.000000" : 34048,
"99.000000" : 51968,
"99.500000" : 53504,
"99.900000" : 81408,
"99.950000" : 96768,
"99.990000" : 113152
}
},
"lat_ns" : {
"min" : 13285,
"max" : 206557,
"mean" : 26496.728306,
"stddev" : 7666.456687,
"N" : 1037222
},
"bw_min" : 591632,
"bw_max" : 593816,
"bw_agg" : 100.000000,
"bw_mean" : 592752.785714,
"bw_dev" : 705.015022,
"bw_samples" : 14,
"iops_min" : 147908,
"iops_max" : 148454,
"iops_mean" : 148188.142857,
"iops_stddev" : 176.173803,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 16.328571,
"sys_cpu" : 55.942857,
"ctx" : 266741,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 45.803249,
"50" : 52.586484,
"100" : 1.563893,
"250" : 0.044157,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=87319: Tue Oct 25 23:39:21 2022
read: IOPS=148k, BW=579MiB/s (607MB/s)(4052MiB/7001msec)
slat (nsec): min=1874, max=27602, avg=2143.57, stdev=413.58
clat (usec): min=2, max=204, avg=24.35, stdev= 7.56
lat (usec): min=13, max=206, avg=26.50, stdev= 7.67
clat percentiles (usec):
| 1.00th=[ 17], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18],
| 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 29], 60.00th=[ 29],
| 70.00th=[ 29], 80.00th=[ 29], 90.00th=[ 30], 95.00th=[ 35],
| 99.00th=[ 52], 99.50th=[ 54], 99.90th=[ 82], 99.95th=[ 97],
| 99.99th=[ 114]
bw ( KiB/s): min=591632, max=593816, per=100.00%, avg=592752.79, stdev=705.02, samples=14
iops : min=147908, max=148454, avg=148188.14, stdev=176.17, samples=14
lat (usec) : 4=0.01%, 10=0.01%, 20=45.80%, 50=52.59%, 100=1.56%
lat (usec) : 250=0.04%
cpu : usr=16.33%, sys=55.94%, ctx=266741, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1037219,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=579MiB/s (607MB/s), 579MiB/s-579MiB/s (607MB/s-607MB/s), io=4052MiB (4248MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt"
$ cat /tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733974,
"timestamp_ms" : 1666733974770,
"time" : "Tue Oct 25 23:39:34 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 4263362560,
"io_kbytes" : 4163440,
"bw_bytes" : 608964799,
"bw" : 594692,
"iops" : 148672.618197,
"runtime" : 7001,
"total_ios" : 1040857,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1883,
"max" : 30347,
"mean" : 2119.325331,
"stddev" : 420.060474,
"N" : 1040857
},
"clat_ns" : {
"min" : 251,
"max" : 149521,
"mean" : 24289.765743,
"stddev" : 7494.997171,
"N" : 1040860,
"percentile" : {
"1.000000" : 15168,
"5.000000" : 17024,
"10.000000" : 17024,
"20.000000" : 17792,
"30.000000" : 17792,
"40.000000" : 18560,
"50.000000" : 27776,
"60.000000" : 28032,
"70.000000" : 28288,
"80.000000" : 28800,
"90.000000" : 29056,
"95.000000" : 34048,
"99.000000" : 51968,
"99.500000" : 53504,
"99.900000" : 81408,
"99.950000" : 97792,
"99.990000" : 113152
}
},
"lat_ns" : {
"min" : 14096,
"max" : 151475,
"mean" : 26409.091097,
"stddev" : 7588.653299,
"N" : 1040860
},
"bw_min" : 593880,
"bw_max" : 595576,
"bw_agg" : 100.000000,
"bw_mean" : 594850.642857,
"bw_dev" : 491.431129,
"bw_samples" : 14,
"iops_min" : 148470,
"iops_max" : 148894,
"iops_mean" : 148712.642857,
"iops_stddev" : 122.870543,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.671429,
"sys_cpu" : 58.242857,
"ctx" : 267603,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 45.125891,
"50" : 53.340180,
"100" : 1.487716,
"250" : 0.043714,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=87357: Tue Oct 25 23:39:34 2022
read: IOPS=149k, BW=581MiB/s (609MB/s)(4066MiB/7001msec)
slat (nsec): min=1883, max=30347, avg=2119.33, stdev=420.06
clat (nsec): min=251, max=149521, avg=24289.77, stdev=7495.00
lat (usec): min=14, max=151, avg=26.41, stdev= 7.59
clat percentiles (usec):
| 1.00th=[ 16], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18],
| 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 28], 60.00th=[ 29],
| 70.00th=[ 29], 80.00th=[ 29], 90.00th=[ 30], 95.00th=[ 35],
| 99.00th=[ 52], 99.50th=[ 54], 99.90th=[ 82], 99.95th=[ 98],
| 99.99th=[ 114]
bw ( KiB/s): min=593880, max=595576, per=100.00%, avg=594850.64, stdev=491.43, samples=14
iops : min=148470, max=148894, avg=148712.64, stdev=122.87, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=45.13%, 50=53.34%, 100=1.49%
lat (usec) : 250=0.04%
cpu : usr=13.67%, sys=58.24%, ctx=267603, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1040857,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=581MiB/s (609MB/s), 581MiB/s-581MiB/s (609MB/s-609MB/s), io=4066MiB (4263MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt"
$ cat /tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733085,
"timestamp_ms" : 1666733085963,
"time" : "Tue Oct 25 23:24:45 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2405982208,
"io_kbytes" : 2349592,
"bw_bytes" : 343662649,
"bw" : 335608,
"iops" : 83902.013998,
"runtime" : 7001,
"total_ios" : 587398,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 711,
"max" : 4368,
"mean" : 924.473922,
"stddev" : 69.051652,
"N" : 587398
},
"clat_ns" : {
"min" : 9077,
"max" : 113724,
"mean" : 10706.197505,
"stddev" : 2217.342043,
"N" : 587398,
"percentile" : {
"1.000000" : 9920,
"5.000000" : 10048,
"10.000000" : 10048,
"20.000000" : 10176,
"30.000000" : 10304,
"40.000000" : 10304,
"50.000000" : 10432,
"60.000000" : 10432,
"70.000000" : 10560,
"80.000000" : 10560,
"90.000000" : 10688,
"95.000000" : 11072,
"99.000000" : 20096,
"99.500000" : 27520,
"99.900000" : 30080,
"99.950000" : 48384,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 10510,
"max" : 114625,
"mean" : 11630.671427,
"stddev" : 2218.545088,
"N" : 587398
},
"bw_min" : 335280,
"bw_max" : 336168,
"bw_agg" : 100.000000,
"bw_mean" : 335695.428571,
"bw_dev" : 291.592369,
"bw_samples" : 14,
"iops_min" : 83820,
"iops_max" : 84042,
"iops_mean" : 83923.857143,
"iops_stddev" : 72.898092,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 16,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 3.081897,
"20" : 95.920653,
"50" : 0.950122,
"100" : 0.047157,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=84633: Tue Oct 25 23:24:45 2022
read: IOPS=83.9k, BW=328MiB/s (344MB/s)(2295MiB/7001msec)
slat (nsec): min=711, max=4368, avg=924.47, stdev=69.05
clat (usec): min=9, max=113, avg=10.71, stdev= 2.22
lat (usec): min=10, max=114, avg=11.63, stdev= 2.22
clat percentiles (nsec):
| 1.00th=[ 9920], 5.00th=[10048], 10.00th=[10048], 20.00th=[10176],
| 30.00th=[10304], 40.00th=[10304], 50.00th=[10432], 60.00th=[10432],
| 70.00th=[10560], 80.00th=[10560], 90.00th=[10688], 95.00th=[11072],
| 99.00th=[20096], 99.50th=[27520], 99.90th=[30080], 99.95th=[48384],
| 99.99th=[68096]
bw ( KiB/s): min=335280, max=336168, per=100.00%, avg=335695.43, stdev=291.59, samples=14
iops : min=83820, max=84042, avg=83923.86, stdev=72.90, samples=14
lat (usec) : 10=3.08%, 20=95.92%, 50=0.95%, 100=0.05%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=16, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=587398,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=2295MiB (2406MB), run=7001-7001msec
$ cat /tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733988,
"timestamp_ms" : 1666733988393,
"time" : "Tue Oct 25 23:39:48 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 4244062208,
"io_kbytes" : 4144592,
"bw_bytes" : 606208000,
"bw" : 592000,
"iops" : 147999.428653,
"runtime" : 7001,
"total_ios" : 1036144,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1973,
"max" : 32401,
"mean" : 2217.550113,
"stddev" : 422.893349,
"N" : 1036145
},
"clat_ns" : {
"min" : 260,
"max" : 149982,
"mean" : 24348.226217,
"stddev" : 7711.887494,
"N" : 1036147,
"percentile" : {
"1.000000" : 16512,
"5.000000" : 16768,
"10.000000" : 16768,
"20.000000" : 17536,
"30.000000" : 17536,
"40.000000" : 18304,
"50.000000" : 28288,
"60.000000" : 28288,
"70.000000" : 28544,
"80.000000" : 29056,
"90.000000" : 29312,
"95.000000" : 34048,
"99.000000" : 51968,
"99.500000" : 53504,
"99.900000" : 80384,
"99.950000" : 96768,
"99.990000" : 113152
}
},
"lat_ns" : {
"min" : 13815,
"max" : 152456,
"mean" : 26565.776468,
"stddev" : 7822.726275,
"N" : 1036147
},
"bw_min" : 590720,
"bw_max" : 593480,
"bw_agg" : 100.000000,
"bw_mean" : 592140.500000,
"bw_dev" : 670.367379,
"bw_samples" : 14,
"iops_min" : 147680,
"iops_max" : 148370,
"iops_mean" : 148035.071429,
"iops_stddev" : 167.612403,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.842857,
"sys_cpu" : 56.142857,
"ctx" : 265858,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 45.727042,
"50" : 52.683121,
"100" : 1.546310,
"250" : 0.040824,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=87395: Tue Oct 25 23:39:48 2022
read: IOPS=148k, BW=578MiB/s (606MB/s)(4047MiB/7001msec)
slat (nsec): min=1973, max=32401, avg=2217.55, stdev=422.89
clat (nsec): min=260, max=149982, avg=24348.23, stdev=7711.89
lat (usec): min=13, max=152, avg=26.57, stdev= 7.82
clat percentiles (usec):
| 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 18],
| 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 29], 60.00th=[ 29],
| 70.00th=[ 29], 80.00th=[ 30], 90.00th=[ 30], 95.00th=[ 35],
| 99.00th=[ 52], 99.50th=[ 54], 99.90th=[ 81], 99.95th=[ 97],
| 99.99th=[ 114]
bw ( KiB/s): min=590720, max=593480, per=100.00%, avg=592140.50, stdev=670.37, samples=14
iops : min=147680, max=148370, avg=148035.07, stdev=167.61, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=45.73%, 50=52.68%, 100=1.55%
lat (usec) : 250=0.04%
cpu : usr=17.84%, sys=56.14%, ctx=265858, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1036144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=578MiB/s (606MB/s), 578MiB/s-578MiB/s (606MB/s-606MB/s), io=4047MiB (4244MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt"
$ cat /tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734002,
"timestamp_ms" : 1666734002404,
"time" : "Tue Oct 25 23:40:02 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 4233482240,
"io_kbytes" : 4134260,
"bw_bytes" : 604696791,
"bw" : 590524,
"iops" : 147630.624197,
"runtime" : 7001,
"total_ios" : 1033562,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1973,
"max" : 28543,
"mean" : 2227.234882,
"stddev" : 433.965898,
"N" : 1033562
},
"clat_ns" : {
"min" : 291,
"max" : 150883,
"mean" : 24397.982275,
"stddev" : 7660.972958,
"N" : 1033565,
"percentile" : {
"1.000000" : 16512,
"5.000000" : 16768,
"10.000000" : 16768,
"20.000000" : 17536,
"30.000000" : 17792,
"40.000000" : 18304,
"50.000000" : 28288,
"60.000000" : 28544,
"70.000000" : 28544,
"80.000000" : 29056,
"90.000000" : 29312,
"95.000000" : 34048,
"99.000000" : 51968,
"99.500000" : 52992,
"99.900000" : 76288,
"99.950000" : 94720,
"99.990000" : 112128
}
},
"lat_ns" : {
"min" : 13616,
"max" : 153037,
"mean" : 26625.217129,
"stddev" : 7783.374398,
"N" : 1033565
},
"bw_min" : 589488,
"bw_max" : 591486,
"bw_agg" : 100.000000,
"bw_mean" : 590663.285714,
"bw_dev" : 586.088787,
"bw_samples" : 14,
"iops_min" : 147372,
"iops_max" : 147871,
"iops_mean" : 147665.785714,
"iops_stddev" : 146.583753,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 18.671429,
"sys_cpu" : 55.157143,
"ctx" : 265038,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 45.679117,
"50" : 52.747102,
"100" : 1.533628,
"250" : 0.037540,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=87435: Tue Oct 25 23:40:02 2022
read: IOPS=148k, BW=577MiB/s (605MB/s)(4037MiB/7001msec)
slat (nsec): min=1973, max=28543, avg=2227.23, stdev=433.97
clat (nsec): min=291, max=150883, avg=24397.98, stdev=7660.97
lat (usec): min=13, max=153, avg=26.63, stdev= 7.78
clat percentiles (usec):
| 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 18],
| 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 29], 60.00th=[ 29],
| 70.00th=[ 29], 80.00th=[ 30], 90.00th=[ 30], 95.00th=[ 35],
| 99.00th=[ 52], 99.50th=[ 53], 99.90th=[ 77], 99.95th=[ 95],
| 99.99th=[ 113]
bw ( KiB/s): min=589488, max=591486, per=100.00%, avg=590663.29, stdev=586.09, samples=14
iops : min=147372, max=147871, avg=147665.79, stdev=146.58, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=45.68%, 50=52.75%, 100=1.53%
lat (usec) : 250=0.04%
cpu : usr=18.67%, sys=55.16%, ctx=265038, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1033562,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=577MiB/s (605MB/s), 577MiB/s-577MiB/s (605MB/s-605MB/s), io=4037MiB (4233MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt"
$ cat /tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734016,
"timestamp_ms" : 1666734016263,
"time" : "Tue Oct 25 23:40:16 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 4226142208,
"io_kbytes" : 4127092,
"bw_bytes" : 603648365,
"bw" : 589500,
"iops" : 147374.517926,
"runtime" : 7001,
"total_ios" : 1031769,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1993,
"max" : 26640,
"mean" : 2267.380921,
"stddev" : 434.200456,
"N" : 1031770
},
"clat_ns" : {
"min" : 310,
"max" : 200466,
"mean" : 24390.233048,
"stddev" : 7872.725576,
"N" : 1031772,
"percentile" : {
"1.000000" : 16192,
"5.000000" : 16512,
"10.000000" : 16768,
"20.000000" : 17280,
"30.000000" : 17536,
"40.000000" : 18304,
"50.000000" : 28288,
"60.000000" : 28544,
"70.000000" : 28800,
"80.000000" : 29056,
"90.000000" : 29568,
"95.000000" : 34048,
"99.000000" : 51968,
"99.500000" : 53504,
"99.900000" : 84480,
"99.950000" : 99840,
"99.990000" : 115200
}
},
"lat_ns" : {
"min" : 13556,
"max" : 203111,
"mean" : 26657.614235,
"stddev" : 7990.434757,
"N" : 1031772
},
"bw_min" : 587911,
"bw_max" : 591096,
"bw_agg" : 100.000000,
"bw_mean" : 589642.214286,
"bw_dev" : 928.622982,
"bw_samples" : 14,
"iops_min" : 146977,
"iops_max" : 147774,
"iops_mean" : 147410.500000,
"iops_stddev" : 232.283232,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 18.514286,
"sys_cpu" : 56.057143,
"ctx" : 264480,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 45.311111,
"50" : 53.090178,
"100" : 1.546858,
"250" : 0.049139,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=87473: Tue Oct 25 23:40:16 2022
read: IOPS=147k, BW=576MiB/s (604MB/s)(4030MiB/7001msec)
slat (nsec): min=1993, max=26640, avg=2267.38, stdev=434.20
clat (nsec): min=310, max=200466, avg=24390.23, stdev=7872.73
lat (usec): min=13, max=203, avg=26.66, stdev= 7.99
clat percentiles (usec):
| 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 18],
| 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 29], 60.00th=[ 29],
| 70.00th=[ 29], 80.00th=[ 30], 90.00th=[ 30], 95.00th=[ 35],
| 99.00th=[ 52], 99.50th=[ 54], 99.90th=[ 85], 99.95th=[ 100],
| 99.99th=[ 116]
bw ( KiB/s): min=587911, max=591096, per=100.00%, avg=589642.21, stdev=928.62, samples=14
iops : min=146977, max=147774, avg=147410.50, stdev=232.28, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=45.31%, 50=53.09%, 100=1.55%
lat (usec) : 250=0.05%
cpu : usr=18.51%, sys=56.06%, ctx=264480, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1031769,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=576MiB/s (604MB/s), 576MiB/s-576MiB/s (604MB/s-604MB/s), io=4030MiB (4226MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734031,
"timestamp_ms" : 1666734031180,
"time" : "Tue Oct 25 23:40:31 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 5484310528,
"io_kbytes" : 5355772,
"bw_bytes" : 783361023,
"bw" : 765000,
"iops" : 191249.821454,
"runtime" : 7001,
"total_ios" : 1338940,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 661,
"max" : 38161,
"mean" : 943.770155,
"stddev" : 157.273061,
"N" : 1338940
},
"clat_ns" : {
"min" : 1282,
"max" : 194796,
"mean" : 19434.049774,
"stddev" : 6553.654058,
"N" : 1338942,
"percentile" : {
"1.000000" : 11200,
"5.000000" : 12480,
"10.000000" : 12992,
"20.000000" : 15936,
"30.000000" : 16320,
"40.000000" : 17024,
"50.000000" : 18048,
"60.000000" : 19328,
"70.000000" : 20352,
"80.000000" : 22656,
"90.000000" : 24448,
"95.000000" : 32640,
"99.000000" : 44288,
"99.500000" : 47360,
"99.900000" : 77312,
"99.950000" : 92672,
"99.990000" : 105984
}
},
"lat_ns" : {
"min" : 11071,
"max" : 195707,
"mean" : 20377.819873,
"stddev" : 6563.731917,
"N" : 1338942
},
"bw_min" : 764144,
"bw_max" : 766368,
"bw_agg" : 100.000000,
"bw_mean" : 765234.071429,
"bw_dev" : 634.098083,
"bw_samples" : 14,
"iops_min" : 191036,
"iops_max" : 191592,
"iops_mean" : 191308.357143,
"iops_stddev" : 158.356757,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 9,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 19.636429,
"4" : 80.350725,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 65.775315,
"50" : 33.972023,
"100" : 0.223236,
"250" : 0.027634,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87522: Tue Oct 25 23:40:31 2022
read: IOPS=191k, BW=747MiB/s (783MB/s)(5230MiB/7001msec)
slat (nsec): min=661, max=38161, avg=943.77, stdev=157.27
clat (nsec): min=1282, max=194796, avg=19434.05, stdev=6553.65
lat (usec): min=11, max=195, avg=20.38, stdev= 6.56
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 16],
| 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 20],
| 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 25], 95.00th=[ 33],
| 99.00th=[ 45], 99.50th=[ 48], 99.90th=[ 78], 99.95th=[ 93],
| 99.99th=[ 106]
bw ( KiB/s): min=764144, max=766368, per=100.00%, avg=765234.07, stdev=634.10, samples=14
iops : min=191036, max=191592, avg=191308.36, stdev=158.36, samples=14
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=65.78%, 50=33.97%
lat (usec) : 100=0.22%, 250=0.03%
cpu : usr=99.99%, sys=0.00%, ctx=9, majf=0, minf=0
IO depths : 1=0.1%, 2=19.6%, 4=80.4%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1338940,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=747MiB/s (783MB/s), 747MiB/s-747MiB/s (783MB/s-783MB/s), io=5230MiB (5484MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory
$ rm /tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734046,
"timestamp_ms" : 1666734046275,
"time" : "Tue Oct 25 23:40:46 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 5460344832,
"io_kbytes" : 5332368,
"bw_bytes" : 779937842,
"bw" : 761658,
"iops" : 190414.083702,
"runtime" : 7001,
"total_ios" : 1333089,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 651,
"max" : 38282,
"mean" : 939.539594,
"stddev" : 143.340985,
"N" : 1333089
},
"clat_ns" : {
"min" : 1283,
"max" : 151294,
"mean" : 19526.900070,
"stddev" : 6576.455932,
"N" : 1333092,
"percentile" : {
"1.000000" : 11328,
"5.000000" : 12480,
"10.000000" : 13120,
"20.000000" : 16064,
"30.000000" : 16512,
"40.000000" : 17024,
"50.000000" : 18048,
"60.000000" : 19328,
"70.000000" : 20352,
"80.000000" : 22912,
"90.000000" : 24448,
"95.000000" : 32640,
"99.000000" : 44288,
"99.500000" : 47872,
"99.900000" : 79360,
"99.950000" : 93696,
"99.990000" : 105984
}
},
"lat_ns" : {
"min" : 10961,
"max" : 152466,
"mean" : 20466.439579,
"stddev" : 6586.533800,
"N" : 1333092
},
"bw_min" : 760152,
"bw_max" : 764288,
"bw_agg" : 100.000000,
"bw_mean" : 761844.357143,
"bw_dev" : 972.547456,
"bw_samples" : 14,
"iops_min" : 190038,
"iops_max" : 191072,
"iops_mean" : 190461.071429,
"iops_stddev" : 243.120883,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 19,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 19.228874,
"4" : 80.757024,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 65.134736,
"50" : 34.590939,
"100" : 0.244395,
"250" : 0.028730,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87573: Tue Oct 25 23:40:46 2022
read: IOPS=190k, BW=744MiB/s (780MB/s)(5207MiB/7001msec)
slat (nsec): min=651, max=38282, avg=939.54, stdev=143.34
clat (nsec): min=1283, max=151294, avg=19526.90, stdev=6576.46
lat (usec): min=10, max=152, avg=20.47, stdev= 6.59
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 17],
| 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 20],
| 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 25], 95.00th=[ 33],
| 99.00th=[ 45], 99.50th=[ 48], 99.90th=[ 80], 99.95th=[ 94],
| 99.99th=[ 106]
bw ( KiB/s): min=760152, max=764288, per=100.00%, avg=761844.36, stdev=972.55, samples=14
iops : min=190038, max=191072, avg=190461.07, stdev=243.12, samples=14
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=65.13%, 50=34.59%
lat (usec) : 100=0.24%, 250=0.03%
cpu : usr=99.99%, sys=0.00%, ctx=19, majf=0, minf=0
IO depths : 1=0.1%, 2=19.2%, 4=80.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1333089,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=744MiB/s (780MB/s), 744MiB/s-744MiB/s (780MB/s-780MB/s), io=5207MiB (5460MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734061,
"timestamp_ms" : 1666734061334,
"time" : "Tue Oct 25 23:41:01 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 5464141824,
"io_kbytes" : 5336076,
"bw_bytes" : 780480191,
"bw" : 762187,
"iops" : 190546.493358,
"runtime" : 7001,
"total_ios" : 1334016,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 661,
"max" : 14577,
"mean" : 940.799248,
"stddev" : 97.471830,
"N" : 1334016
},
"clat_ns" : {
"min" : 9919,
"max" : 193162,
"mean" : 19508.399007,
"stddev" : 6583.473141,
"N" : 1334019,
"percentile" : {
"1.000000" : 11328,
"5.000000" : 12480,
"10.000000" : 13120,
"20.000000" : 15936,
"30.000000" : 16512,
"40.000000" : 17024,
"50.000000" : 18048,
"60.000000" : 19328,
"70.000000" : 20352,
"80.000000" : 22912,
"90.000000" : 24448,
"95.000000" : 32640,
"99.000000" : 44288,
"99.500000" : 47872,
"99.900000" : 79360,
"99.950000" : 92672,
"99.990000" : 107008
}
},
"lat_ns" : {
"min" : 10850,
"max" : 193833,
"mean" : 20449.198191,
"stddev" : 6592.555887,
"N" : 1334019
},
"bw_min" : 760784,
"bw_max" : 763832,
"bw_agg" : 100.000000,
"bw_mean" : 762373.571429,
"bw_dev" : 891.857932,
"bw_samples" : 14,
"iops_min" : 190196,
"iops_max" : 190958,
"iops_mean" : 190593.357143,
"iops_stddev" : 222.929312,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 18,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 19.315660,
"4" : 80.683590,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 65.097870,
"50" : 34.639989,
"100" : 0.232681,
"250" : 0.029610,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87623: Tue Oct 25 23:41:01 2022
read: IOPS=191k, BW=744MiB/s (780MB/s)(5211MiB/7001msec)
slat (nsec): min=661, max=14577, avg=940.80, stdev=97.47
clat (usec): min=9, max=193, avg=19.51, stdev= 6.58
lat (usec): min=10, max=193, avg=20.45, stdev= 6.59
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 16],
| 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 20],
| 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 25], 95.00th=[ 33],
| 99.00th=[ 45], 99.50th=[ 48], 99.90th=[ 80], 99.95th=[ 93],
| 99.99th=[ 108]
bw ( KiB/s): min=760784, max=763832, per=100.00%, avg=762373.57, stdev=891.86, samples=14
iops : min=190196, max=190958, avg=190593.36, stdev=222.93, samples=14
lat (usec) : 10=0.01%, 20=65.10%, 50=34.64%, 100=0.23%, 250=0.03%
cpu : usr=99.99%, sys=0.00%, ctx=18, majf=0, minf=0
IO depths : 1=0.1%, 2=19.3%, 4=80.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1334016,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=744MiB/s (780MB/s), 744MiB/s-744MiB/s (780MB/s-780MB/s), io=5211MiB (5464MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-reference_0.txt"
$ cat /tmp/fio-output_4K_4_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734075,
"timestamp_ms" : 1666734075058,
"time" : "Tue Oct 25 23:41:15 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 4150661120,
"io_kbytes" : 4053380,
"bw_bytes" : 592866893,
"bw" : 578971,
"iops" : 144742.465362,
"runtime" : 7001,
"total_ios" : 1013342,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2234,
"max" : 35627,
"mean" : 2427.005221,
"stddev" : 421.685182,
"N" : 1013342
},
"clat_ns" : {
"min" : 160,
"max" : 159249,
"mean" : 24692.870506,
"stddev" : 8160.319348,
"N" : 1013345,
"percentile" : {
"1.000000" : 14400,
"5.000000" : 14528,
"10.000000" : 14912,
"20.000000" : 16768,
"30.000000" : 17536,
"40.000000" : 20864,
"50.000000" : 28800,
"60.000000" : 29056,
"70.000000" : 29312,
"80.000000" : 29824,
"90.000000" : 30080,
"95.000000" : 34560,
"99.000000" : 52992,
"99.500000" : 55552,
"99.900000" : 80384,
"99.950000" : 96768,
"99.990000" : 113152
}
},
"lat_ns" : {
"min" : 13666,
"max" : 161533,
"mean" : 27119.875680,
"stddev" : 8241.903875,
"N" : 1013345
},
"bw_min" : 577330,
"bw_max" : 581224,
"bw_agg" : 100.000000,
"bw_mean" : 579190.428571,
"bw_dev" : 1221.195424,
"bw_samples" : 14,
"iops_min" : 144332,
"iops_max" : 145306,
"iops_mean" : 144797.571429,
"iops_stddev" : 305.357474,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.071429,
"sys_cpu" : 60.985714,
"ctx" : 259323,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 35.179239,
"50" : 63.216762,
"100" : 1.556039,
"250" : 0.043223,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 1452241,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 21892,
"write_ticks" : 0,
"in_queue" : 21892,
"util" : 99.049128
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=87662: Tue Oct 25 23:41:15 2022
read: IOPS=145k, BW=565MiB/s (593MB/s)(3958MiB/7001msec)
slat (nsec): min=2234, max=35627, avg=2427.01, stdev=421.69
clat (nsec): min=160, max=159249, avg=24692.87, stdev=8160.32
lat (usec): min=13, max=161, avg=27.12, stdev= 8.24
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 17],
| 30.00th=[ 18], 40.00th=[ 21], 50.00th=[ 29], 60.00th=[ 30],
| 70.00th=[ 30], 80.00th=[ 30], 90.00th=[ 31], 95.00th=[ 35],
| 99.00th=[ 53], 99.50th=[ 56], 99.90th=[ 81], 99.95th=[ 97],
| 99.99th=[ 114]
bw ( KiB/s): min=577330, max=581224, per=100.00%, avg=579190.43, stdev=1221.20, samples=14
iops : min=144332, max=145306, avg=144797.57, stdev=305.36, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=35.18%, 50=63.22%, 100=1.56%
lat (usec) : 250=0.04%
cpu : usr=15.07%, sys=60.99%, ctx=259323, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1013342,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=565MiB/s (593MB/s), 565MiB/s-565MiB/s (593MB/s-593MB/s), io=3958MiB (4151MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=1452241/0, merge=0/0, ticks=21892/0, in_queue=21892, util=99.05%
$ rm /tmp/fio-output_4K_4_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-reference_1.txt"
$ cat /tmp/fio-output_4K_4_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734088,
"timestamp_ms" : 1666734088701,
"time" : "Tue Oct 25 23:41:28 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 4203417600,
"io_kbytes" : 4104900,
"bw_bytes" : 600402456,
"bw" : 586330,
"iops" : 146582.202542,
"runtime" : 7001,
"total_ios" : 1026222,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2174,
"max" : 178164,
"mean" : 2386.173347,
"stddev" : 455.786112,
"N" : 1026222
},
"clat_ns" : {
"min" : 191,
"max" : 208732,
"mean" : 24370.688623,
"stddev" : 8077.609207,
"N" : 1026225,
"percentile" : {
"1.000000" : 14272,
"5.000000" : 15040,
"10.000000" : 16512,
"20.000000" : 16768,
"30.000000" : 17280,
"40.000000" : 17792,
"50.000000" : 28544,
"60.000000" : 28800,
"70.000000" : 28800,
"80.000000" : 29056,
"90.000000" : 29568,
"95.000000" : 34560,
"99.000000" : 52480,
"99.500000" : 54016,
"99.900000" : 83456,
"99.950000" : 99840,
"99.990000" : 115200
}
},
"lat_ns" : {
"min" : 11301,
"max" : 217257,
"mean" : 26756.861984,
"stddev" : 8171.492717,
"N" : 1026225
},
"bw_min" : 584704,
"bw_max" : 588312,
"bw_agg" : 100.000000,
"bw_mean" : 586556.714286,
"bw_dev" : 891.194395,
"bw_samples" : 14,
"iops_min" : 146176,
"iops_max" : 147078,
"iops_mean" : 146639.000000,
"iops_stddev" : 222.618681,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.600000,
"sys_cpu" : 60.242857,
"ctx" : 262841,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 41.038976,
"50" : 57.387485,
"100" : 1.520529,
"250" : 0.049210,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 1469749,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 22115,
"write_ticks" : 0,
"in_queue" : 22115,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=87701: Tue Oct 25 23:41:28 2022
read: IOPS=147k, BW=573MiB/s (600MB/s)(4009MiB/7001msec)
slat (nsec): min=2174, max=178164, avg=2386.17, stdev=455.79
clat (nsec): min=191, max=208732, avg=24370.69, stdev=8077.61
lat (usec): min=11, max=217, avg=26.76, stdev= 8.17
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 17],
| 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 29], 60.00th=[ 29],
| 70.00th=[ 29], 80.00th=[ 30], 90.00th=[ 30], 95.00th=[ 35],
| 99.00th=[ 53], 99.50th=[ 55], 99.90th=[ 84], 99.95th=[ 100],
| 99.99th=[ 116]
bw ( KiB/s): min=584704, max=588312, per=100.00%, avg=586556.71, stdev=891.19, samples=14
iops : min=146176, max=147078, avg=146639.00, stdev=222.62, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=41.04%, 50=57.39%, 100=1.52%
lat (usec) : 250=0.05%
cpu : usr=15.60%, sys=60.24%, ctx=262841, majf=0, minf=1
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1026222,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=573MiB/s (600MB/s), 573MiB/s-573MiB/s (600MB/s-600MB/s), io=4009MiB (4203MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=1469749/0, merge=0/0, ticks=22115/0, in_queue=22115, util=99.09%
$ rm /tmp/fio-output_4K_4_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-reference_2.txt"
$ cat /tmp/fio-output_4K_4_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734102,
"timestamp_ms" : 1666734102332,
"time" : "Tue Oct 25 23:41:42 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 4528324608,
"io_kbytes" : 4422192,
"bw_bytes" : 646811113,
"bw" : 631651,
"iops" : 157912.441080,
"runtime" : 7001,
"total_ios" : 1105545,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2275,
"max" : 119033,
"mean" : 2745.671780,
"stddev" : 595.147245,
"N" : 1105545
},
"clat_ns" : {
"min" : 3747,
"max" : 138570,
"mean" : 22051.316446,
"stddev" : 5281.925427,
"N" : 1105548,
"percentile" : {
"1.000000" : 17792,
"5.000000" : 19328,
"10.000000" : 19328,
"20.000000" : 19840,
"30.000000" : 20864,
"40.000000" : 20864,
"50.000000" : 21120,
"60.000000" : 21120,
"70.000000" : 21376,
"80.000000" : 21632,
"90.000000" : 22400,
"95.000000" : 33024,
"99.000000" : 47360,
"99.500000" : 50432,
"99.900000" : 79360,
"99.950000" : 93696,
"99.990000" : 105984
}
},
"lat_ns" : {
"min" : 12774,
"max" : 140935,
"mean" : 24796.987255,
"stddev" : 5227.303506,
"N" : 1105548
},
"bw_min" : 630016,
"bw_max" : 633688,
"bw_agg" : 100.000000,
"bw_mean" : 632033.142857,
"bw_dev" : 1130.843651,
"bw_samples" : 14,
"iops_min" : 157504,
"iops_max" : 158422,
"iops_mean" : 158008.214286,
"iops_stddev" : 282.708923,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.085714,
"sys_cpu" : 57.100000,
"ctx" : 284692,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 21.244273,
"50" : 78.183339,
"100" : 0.536839,
"250" : 0.034915,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 1564440,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 23344,
"write_ticks" : 0,
"in_queue" : 23344,
"util" : 99.108381
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=87740: Tue Oct 25 23:41:42 2022
read: IOPS=158k, BW=617MiB/s (647MB/s)(4319MiB/7001msec)
slat (nsec): min=2275, max=119033, avg=2745.67, stdev=595.15
clat (usec): min=3, max=138, avg=22.05, stdev= 5.28
lat (usec): min=12, max=140, avg=24.80, stdev= 5.23
clat percentiles (usec):
| 1.00th=[ 18], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 20],
| 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 22],
| 70.00th=[ 22], 80.00th=[ 22], 90.00th=[ 23], 95.00th=[ 34],
| 99.00th=[ 48], 99.50th=[ 51], 99.90th=[ 80], 99.95th=[ 94],
| 99.99th=[ 106]
bw ( KiB/s): min=630016, max=633688, per=100.00%, avg=632033.14, stdev=1130.84, samples=14
iops : min=157504, max=158422, avg=158008.21, stdev=282.71, samples=14
lat (usec) : 4=0.01%, 10=0.01%, 20=21.24%, 50=78.18%, 100=0.54%
lat (usec) : 250=0.03%
cpu : usr=14.09%, sys=57.10%, ctx=284692, majf=0, minf=1
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1105545,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=617MiB/s (647MB/s), 617MiB/s-617MiB/s (647MB/s-647MB/s), io=4319MiB (4528MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=1564440/0, merge=0/0, ticks=23344/0, in_queue=23344, util=99.11%
$ rm /tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734117,
"timestamp_ms" : 1666734117300,
"time" : "Tue Oct 25 23:41:57 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 5844619264,
"io_kbytes" : 5707636,
"bw_bytes" : 834826348,
"bw" : 815260,
"iops" : 203814.597915,
"runtime" : 7001,
"total_ios" : 1426906,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 15208,
"mean" : 166.282201,
"stddev" : 53.441501,
"N" : 1426906
},
"clat_ns" : {
"min" : 10440,
"max" : 145372,
"mean" : 19068.696730,
"stddev" : 5673.395713,
"N" : 1426908,
"percentile" : {
"1.000000" : 15424,
"5.000000" : 16320,
"10.000000" : 17024,
"20.000000" : 17536,
"30.000000" : 17536,
"40.000000" : 17536,
"50.000000" : 17792,
"60.000000" : 17792,
"70.000000" : 18048,
"80.000000" : 18048,
"90.000000" : 19584,
"95.000000" : 31104,
"99.000000" : 43776,
"99.500000" : 46336,
"99.900000" : 83456,
"99.950000" : 98816,
"99.990000" : 123392
}
},
"lat_ns" : {
"min" : 10620,
"max" : 145553,
"mean" : 19234.978923,
"stddev" : 5673.110075,
"N" : 1426908
},
"bw_min" : 812432,
"bw_max" : 821851,
"bw_agg" : 100.000000,
"bw_mean" : 815661.571429,
"bw_dev" : 2674.021567,
"bw_samples" : 14,
"iops_min" : 203108,
"iops_max" : 205462,
"iops_mean" : 203915.428571,
"iops_stddev" : 668.389306,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 60.214286,
"sys_cpu" : 39.757143,
"ctx" : 14,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 30.973379,
"4" : 69.014077,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 91.161156,
"50" : 8.538124,
"100" : 0.253556,
"250" : 0.047305,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=87788: Tue Oct 25 23:41:57 2022
read: IOPS=204k, BW=796MiB/s (835MB/s)(5574MiB/7001msec)
slat (nsec): min=110, max=15208, avg=166.28, stdev=53.44
clat (usec): min=10, max=145, avg=19.07, stdev= 5.67
lat (usec): min=10, max=145, avg=19.23, stdev= 5.67
clat percentiles (usec):
| 1.00th=[ 16], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 18],
| 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 18], 60.00th=[ 18],
| 70.00th=[ 19], 80.00th=[ 19], 90.00th=[ 20], 95.00th=[ 32],
| 99.00th=[ 44], 99.50th=[ 47], 99.90th=[ 84], 99.95th=[ 99],
| 99.99th=[ 124]
bw ( KiB/s): min=812432, max=821851, per=100.00%, avg=815661.57, stdev=2674.02, samples=14
iops : min=203108, max=205462, avg=203915.43, stdev=668.39, samples=14
lat (usec) : 20=91.16%, 50=8.54%, 100=0.25%, 250=0.05%
cpu : usr=60.21%, sys=39.76%, ctx=14, majf=0, minf=0
IO depths : 1=0.1%, 2=31.0%, 4=69.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1426906,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=796MiB/s (835MB/s), 796MiB/s-796MiB/s (835MB/s-835MB/s), io=5574MiB (5845MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733101,
"timestamp_ms" : 1666733101115,
"time" : "Tue Oct 25 23:25:01 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2411470848,
"io_kbytes" : 2354952,
"bw_bytes" : 344446628,
"bw" : 336373,
"iops" : 84093.415226,
"runtime" : 7001,
"total_ios" : 588738,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 711,
"max" : 25067,
"mean" : 928.008393,
"stddev" : 146.987262,
"N" : 588738
},
"clat_ns" : {
"min" : 701,
"max" : 113202,
"mean" : 10713.652358,
"stddev" : 2206.439842,
"N" : 588738,
"percentile" : {
"1.000000" : 9920,
"5.000000" : 10048,
"10.000000" : 10176,
"20.000000" : 10176,
"30.000000" : 10304,
"40.000000" : 10304,
"50.000000" : 10432,
"60.000000" : 10432,
"70.000000" : 10432,
"80.000000" : 10560,
"90.000000" : 10688,
"95.000000" : 11200,
"99.000000" : 20096,
"99.500000" : 27520,
"99.900000" : 30592,
"99.950000" : 43776,
"99.990000" : 67072
}
},
"lat_ns" : {
"min" : 10590,
"max" : 114104,
"mean" : 11641.660751,
"stddev" : 2210.286923,
"N" : 588738
},
"bw_min" : 335992,
"bw_max" : 336976,
"bw_agg" : 100.000000,
"bw_mean" : 336465.214286,
"bw_dev" : 296.234361,
"bw_samples" : 14,
"iops_min" : 83998,
"iops_max" : 84244,
"iops_mean" : 84116.285714,
"iops_stddev" : 74.034741,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 15,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 1.542282,
"20" : 97.451498,
"50" : 0.959000,
"100" : 0.044842,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=84683: Tue Oct 25 23:25:01 2022
read: IOPS=84.1k, BW=328MiB/s (344MB/s)(2300MiB/7001msec)
slat (nsec): min=711, max=25067, avg=928.01, stdev=146.99
clat (nsec): min=701, max=113202, avg=10713.65, stdev=2206.44
lat (usec): min=10, max=114, avg=11.64, stdev= 2.21
clat percentiles (nsec):
| 1.00th=[ 9920], 5.00th=[10048], 10.00th=[10176], 20.00th=[10176],
| 30.00th=[10304], 40.00th=[10304], 50.00th=[10432], 60.00th=[10432],
| 70.00th=[10432], 80.00th=[10560], 90.00th=[10688], 95.00th=[11200],
| 99.00th=[20096], 99.50th=[27520], 99.90th=[30592], 99.95th=[43776],
| 99.99th=[67072]
bw ( KiB/s): min=335992, max=336976, per=100.00%, avg=336465.21, stdev=296.23, samples=14
iops : min=83998, max=84244, avg=84116.29, stdev=74.03, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=1.54%, 20=97.45%, 50=0.96%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=15, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=588738,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=2300MiB (2411MB), run=7001-7001msec
$ cat /tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734132,
"timestamp_ms" : 1666734132411,
"time" : "Tue Oct 25 23:42:12 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6327066624,
"io_kbytes" : 6178776,
"bw_bytes" : 903737555,
"bw" : 882556,
"iops" : 220638.623054,
"runtime" : 7001,
"total_ios" : 1544691,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 17432,
"mean" : 189.121560,
"stddev" : 121.269366,
"N" : 1544691
},
"clat_ns" : {
"min" : 9367,
"max" : 202460,
"mean" : 17679.470528,
"stddev" : 5728.764872,
"N" : 1544694,
"percentile" : {
"1.000000" : 11072,
"5.000000" : 15296,
"10.000000" : 15936,
"20.000000" : 16064,
"30.000000" : 16064,
"40.000000" : 16064,
"50.000000" : 16192,
"60.000000" : 16192,
"70.000000" : 16320,
"80.000000" : 16512,
"90.000000" : 20608,
"95.000000" : 29312,
"99.000000" : 39168,
"99.500000" : 44288,
"99.900000" : 81408,
"99.950000" : 92672,
"99.990000" : 105984
}
},
"lat_ns" : {
"min" : 9678,
"max" : 202640,
"mean" : 17868.592084,
"stddev" : 5727.259130,
"N" : 1544694
},
"bw_min" : 880480,
"bw_max" : 884112,
"bw_agg" : 100.000000,
"bw_mean" : 882771.142857,
"bw_dev" : 946.312914,
"bw_samples" : 14,
"iops_min" : 220120,
"iops_max" : 221028,
"iops_mean" : 220692.928571,
"iops_stddev" : 236.341495,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 42.900000,
"sys_cpu" : 57.071429,
"ctx" : 11,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.363309,
"2" : 2.448580,
"4" : 97.188111,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 89.582124,
"50" : 10.162874,
"100" : 0.218814,
"250" : 0.032369,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=87838: Tue Oct 25 23:42:12 2022
read: IOPS=221k, BW=862MiB/s (904MB/s)(6034MiB/7001msec)
slat (nsec): min=110, max=17432, avg=189.12, stdev=121.27
clat (usec): min=9, max=202, avg=17.68, stdev= 5.73
lat (usec): min=9, max=202, avg=17.87, stdev= 5.73
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17],
| 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 17], 60.00th=[ 17],
| 70.00th=[ 17], 80.00th=[ 17], 90.00th=[ 21], 95.00th=[ 30],
| 99.00th=[ 40], 99.50th=[ 45], 99.90th=[ 82], 99.95th=[ 93],
| 99.99th=[ 106]
bw ( KiB/s): min=880480, max=884112, per=100.00%, avg=882771.14, stdev=946.31, samples=14
iops : min=220120, max=221028, avg=220692.93, stdev=236.34, samples=14
lat (usec) : 10=0.01%, 20=89.58%, 50=10.16%, 100=0.22%, 250=0.03%
cpu : usr=42.90%, sys=57.07%, ctx=11, majf=0, minf=0
IO depths : 1=0.4%, 2=2.4%, 4=97.2%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1544691,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=862MiB/s (904MB/s), 862MiB/s-862MiB/s (904MB/s-904MB/s), io=6034MiB (6327MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734147,
"timestamp_ms" : 1666734147528,
"time" : "Tue Oct 25 23:42:27 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6299238400,
"io_kbytes" : 6151600,
"bw_bytes" : 899762662,
"bw" : 878674,
"iops" : 219668.190259,
"runtime" : 7001,
"total_ios" : 1537897,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 120,
"max" : 16481,
"mean" : 214.052651,
"stddev" : 212.685902,
"N" : 1537897
},
"clat_ns" : {
"min" : 9428,
"max" : 196909,
"mean" : 17698.782211,
"stddev" : 5860.452451,
"N" : 1537900,
"percentile" : {
"1.000000" : 11072,
"5.000000" : 14144,
"10.000000" : 15424,
"20.000000" : 15936,
"30.000000" : 16064,
"40.000000" : 16064,
"50.000000" : 16192,
"60.000000" : 16192,
"70.000000" : 16512,
"80.000000" : 17280,
"90.000000" : 21120,
"95.000000" : 29568,
"99.000000" : 41216,
"99.500000" : 44288,
"99.900000" : 81408,
"99.950000" : 93696,
"99.990000" : 105984
}
},
"lat_ns" : {
"min" : 9678,
"max" : 197099,
"mean" : 17912.834803,
"stddev" : 5855.839323,
"N" : 1537900
},
"bw_min" : 875880,
"bw_max" : 881400,
"bw_agg" : 100.000000,
"bw_mean" : 878894.857143,
"bw_dev" : 1748.287903,
"bw_samples" : 14,
"iops_min" : 218970,
"iops_max" : 220350,
"iops_mean" : 219723.714286,
"iops_stddev" : 437.071976,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 43.414286,
"sys_cpu" : 56.557143,
"ctx" : 12,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.328696,
"2" : 5.216539,
"4" : 94.454765,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 87.246285,
"50" : 12.507144,
"100" : 0.209442,
"250" : 0.031602,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=87889: Tue Oct 25 23:42:27 2022
read: IOPS=220k, BW=858MiB/s (900MB/s)(6007MiB/7001msec)
slat (nsec): min=120, max=16481, avg=214.05, stdev=212.69
clat (usec): min=9, max=196, avg=17.70, stdev= 5.86
lat (usec): min=9, max=197, avg=17.91, stdev= 5.86
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 16],
| 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 17], 60.00th=[ 17],
| 70.00th=[ 17], 80.00th=[ 18], 90.00th=[ 22], 95.00th=[ 30],
| 99.00th=[ 42], 99.50th=[ 45], 99.90th=[ 82], 99.95th=[ 94],
| 99.99th=[ 106]
bw ( KiB/s): min=875880, max=881400, per=100.00%, avg=878894.86, stdev=1748.29, samples=14
iops : min=218970, max=220350, avg=219723.71, stdev=437.07, samples=14
lat (usec) : 10=0.01%, 20=87.25%, 50=12.51%, 100=0.21%, 250=0.03%
cpu : usr=43.41%, sys=56.56%, ctx=12, majf=0, minf=0
IO depths : 1=0.3%, 2=5.2%, 4=94.5%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1537897,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=858MiB/s (900MB/s), 858MiB/s-858MiB/s (900MB/s-900MB/s), io=6007MiB (6299MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734162,
"timestamp_ms" : 1666734162525,
"time" : "Tue Oct 25 23:42:42 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6373732352,
"io_kbytes" : 6224348,
"bw_bytes" : 910403135,
"bw" : 889065,
"iops" : 222265.962005,
"runtime" : 7001,
"total_ios" : 1556084,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1462,
"max" : 37179,
"mean" : 1641.605073,
"stddev" : 199.551002,
"N" : 1556084
},
"clat_ns" : {
"min" : 230,
"max" : 126758,
"mean" : 16115.646010,
"stddev" : 5418.188609,
"N" : 1556087,
"percentile" : {
"1.000000" : 14144,
"5.000000" : 14400,
"10.000000" : 14528,
"20.000000" : 14528,
"30.000000" : 14656,
"40.000000" : 14656,
"50.000000" : 14656,
"60.000000" : 14784,
"70.000000" : 14784,
"80.000000" : 14912,
"90.000000" : 16512,
"95.000000" : 27008,
"99.000000" : 36096,
"99.500000" : 42752,
"99.900000" : 78336,
"99.950000" : 88576,
"99.990000" : 103936
}
},
"lat_ns" : {
"min" : 10790,
"max" : 128381,
"mean" : 17757.251002,
"stddev" : 5418.749748,
"N" : 1556087
},
"bw_min" : 888000,
"bw_max" : 890464,
"bw_agg" : 100.000000,
"bw_mean" : 889269.357143,
"bw_dev" : 942.858143,
"bw_samples" : 14,
"iops_min" : 222000,
"iops_max" : 222616,
"iops_mean" : 222317.285714,
"iops_stddev" : 235.776435,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 18.328571,
"sys_cpu" : 81.642857,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 91.798450,
"50" : 8.025659,
"100" : 0.146200,
"250" : 0.019986,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87938: Tue Oct 25 23:42:42 2022
read: IOPS=222k, BW=868MiB/s (910MB/s)(6078MiB/7001msec)
slat (nsec): min=1462, max=37179, avg=1641.61, stdev=199.55
clat (nsec): min=230, max=126758, avg=16115.65, stdev=5418.19
lat (usec): min=10, max=128, avg=17.76, stdev= 5.42
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15],
| 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 15],
| 70.00th=[ 15], 80.00th=[ 15], 90.00th=[ 17], 95.00th=[ 28],
| 99.00th=[ 37], 99.50th=[ 43], 99.90th=[ 79], 99.95th=[ 89],
| 99.99th=[ 104]
bw ( KiB/s): min=888000, max=890464, per=100.00%, avg=889269.36, stdev=942.86, samples=14
iops : min=222000, max=222616, avg=222317.29, stdev=235.78, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=91.80%, 50=8.03%, 100=0.15%
lat (usec) : 250=0.02%
cpu : usr=18.33%, sys=81.64%, ctx=10, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1556084,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=868MiB/s (910MB/s), 868MiB/s-868MiB/s (910MB/s-910MB/s), io=6078MiB (6374MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734177,
"timestamp_ms" : 1666734177491,
"time" : "Tue Oct 25 23:42:57 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6295175168,
"io_kbytes" : 6147632,
"bw_bytes" : 899182283,
"bw" : 878107,
"iops" : 219526.496215,
"runtime" : 7001,
"total_ios" : 1536905,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1593,
"max" : 27100,
"mean" : 1862.124565,
"stddev" : 155.390262,
"N" : 1536905
},
"clat_ns" : {
"min" : 250,
"max" : 140985,
"mean" : 16110.551588,
"stddev" : 6695.143158,
"N" : 1536908,
"percentile" : {
"1.000000" : 10432,
"5.000000" : 10432,
"10.000000" : 10560,
"20.000000" : 12480,
"30.000000" : 12736,
"40.000000" : 13248,
"50.000000" : 14400,
"60.000000" : 14528,
"70.000000" : 16768,
"80.000000" : 20608,
"90.000000" : 21376,
"95.000000" : 29312,
"99.000000" : 41216,
"99.500000" : 44800,
"99.900000" : 80384,
"99.950000" : 96768,
"99.990000" : 122368
}
},
"lat_ns" : {
"min" : 9629,
"max" : 142838,
"mean" : 17972.676267,
"stddev" : 6706.536732,
"N" : 1536908
},
"bw_min" : 876896,
"bw_max" : 880945,
"bw_agg" : 100.000000,
"bw_mean" : 878672.428571,
"bw_dev" : 1464.370472,
"bw_samples" : 14,
"iops_min" : 219224,
"iops_max" : 220236,
"iops_mean" : 219668.071429,
"iops_stddev" : 366.064405,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.700000,
"sys_cpu" : 82.271429,
"ctx" : 18,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.043334,
"20" : 76.322609,
"50" : 23.389604,
"100" : 0.200533,
"250" : 0.043919,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87987: Tue Oct 25 23:42:57 2022
read: IOPS=220k, BW=858MiB/s (899MB/s)(6004MiB/7001msec)
slat (nsec): min=1593, max=27100, avg=1862.12, stdev=155.39
clat (nsec): min=250, max=140985, avg=16110.55, stdev=6695.14
lat (usec): min=9, max=142, avg=17.97, stdev= 6.71
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 13],
| 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 15], 60.00th=[ 15],
| 70.00th=[ 17], 80.00th=[ 21], 90.00th=[ 22], 95.00th=[ 30],
| 99.00th=[ 42], 99.50th=[ 45], 99.90th=[ 81], 99.95th=[ 97],
| 99.99th=[ 123]
bw ( KiB/s): min=876896, max=880945, per=100.00%, avg=878672.43, stdev=1464.37, samples=14
iops : min=219224, max=220236, avg=219668.07, stdev=366.06, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 10=0.04%, 20=76.32%, 50=23.39%, 100=0.20%, 250=0.04%
cpu : usr=17.70%, sys=82.27%, ctx=18, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1536905,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=858MiB/s (899MB/s), 858MiB/s-858MiB/s (899MB/s-899MB/s), io=6004MiB (6295MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory
$ rm /tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734192,
"timestamp_ms" : 1666734192584,
"time" : "Tue Oct 25 23:43:12 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6370635776,
"io_kbytes" : 6221324,
"bw_bytes" : 909960830,
"bw" : 888633,
"iops" : 222157.834595,
"runtime" : 7001,
"total_ios" : 1555327,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1462,
"max" : 27873,
"mean" : 1640.823853,
"stddev" : 204.088642,
"N" : 1555328
},
"clat_ns" : {
"min" : 240,
"max" : 140985,
"mean" : 16120.417763,
"stddev" : 5496.166937,
"N" : 1555330,
"percentile" : {
"1.000000" : 14144,
"5.000000" : 14400,
"10.000000" : 14528,
"20.000000" : 14528,
"30.000000" : 14656,
"40.000000" : 14656,
"50.000000" : 14656,
"60.000000" : 14784,
"70.000000" : 14784,
"80.000000" : 14912,
"90.000000" : 16512,
"95.000000" : 27008,
"99.000000" : 36096,
"99.500000" : 42752,
"99.900000" : 79360,
"99.950000" : 90624,
"99.990000" : 103936
}
},
"lat_ns" : {
"min" : 10810,
"max" : 142668,
"mean" : 17761.241593,
"stddev" : 5496.744254,
"N" : 1555330
},
"bw_min" : 887288,
"bw_max" : 890128,
"bw_agg" : 100.000000,
"bw_mean" : 888840.714286,
"bw_dev" : 890.705375,
"bw_samples" : 14,
"iops_min" : 221822,
"iops_max" : 222532,
"iops_mean" : 222210.285714,
"iops_stddev" : 222.724193,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 18.785714,
"sys_cpu" : 81.185714,
"ctx" : 13,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.012152,
"20" : 91.790730,
"50" : 8.009377,
"100" : 0.164274,
"250" : 0.023082,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=88037: Tue Oct 25 23:43:12 2022
read: IOPS=222k, BW=868MiB/s (910MB/s)(6076MiB/7001msec)
slat (nsec): min=1462, max=27873, avg=1640.82, stdev=204.09
clat (nsec): min=240, max=140985, avg=16120.42, stdev=5496.17
lat (usec): min=10, max=142, avg=17.76, stdev= 5.50
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15],
| 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 15],
| 70.00th=[ 15], 80.00th=[ 15], 90.00th=[ 17], 95.00th=[ 28],
| 99.00th=[ 37], 99.50th=[ 43], 99.90th=[ 80], 99.95th=[ 91],
| 99.99th=[ 104]
bw ( KiB/s): min=887288, max=890128, per=100.00%, avg=888840.71, stdev=890.71, samples=14
iops : min=221822, max=222532, avg=222210.29, stdev=222.72, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=91.79%, 50=8.01%, 100=0.16%
lat (usec) : 250=0.02%
cpu : usr=18.79%, sys=81.19%, ctx=13, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1555327,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=868MiB/s (910MB/s), 868MiB/s-868MiB/s (910MB/s-910MB/s), io=6076MiB (6371MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-xnvme_0.txt"
$ cat /tmp/fio-output_4K_4_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734206,
"timestamp_ms" : 1666734206301,
"time" : "Tue Oct 25 23:43:26 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 4069163008,
"io_kbytes" : 3973792,
"bw_bytes" : 581225968,
"bw" : 567603,
"iops" : 141900.442794,
"runtime" : 7001,
"total_ios" : 993445,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2274,
"max" : 1343589,
"mean" : 2510.766837,
"stddev" : 1415.574549,
"N" : 993445
},
"clat_ns" : {
"min" : 240,
"max" : 1371542,
"mean" : 25199.312395,
"stddev" : 8781.217777,
"N" : 993448,
"percentile" : {
"1.000000" : 14016,
"5.000000" : 14272,
"10.000000" : 14400,
"20.000000" : 14656,
"30.000000" : 20864,
"40.000000" : 21376,
"50.000000" : 29568,
"60.000000" : 30080,
"70.000000" : 30080,
"80.000000" : 30336,
"90.000000" : 30592,
"95.000000" : 35072,
"99.000000" : 52992,
"99.500000" : 56576,
"99.900000" : 82432,
"99.950000" : 97792,
"99.990000" : 114176
}
},
"lat_ns" : {
"min" : 14237,
"max" : 1373906,
"mean" : 27710.079092,
"stddev" : 8975.141547,
"N" : 993448
},
"bw_min" : 566520,
"bw_max" : 570888,
"bw_agg" : 100.000000,
"bw_mean" : 567735.857143,
"bw_dev" : 1371.074472,
"bw_samples" : 14,
"iops_min" : 141630,
"iops_max" : 142722,
"iops_mean" : 141933.928571,
"iops_stddev" : 342.789118,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 20.185714,
"sys_cpu" : 56.685714,
"ctx" : 253738,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 24.128261,
"50" : 74.194948,
"100" : 1.629783,
"250" : 0.044089,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=88076: Tue Oct 25 23:43:26 2022
read: IOPS=142k, BW=554MiB/s (581MB/s)(3881MiB/7001msec)
slat (usec): min=2, max=1343, avg= 2.51, stdev= 1.42
clat (nsec): min=240, max=1371.5k, avg=25199.31, stdev=8781.22
lat (usec): min=14, max=1373, avg=27.71, stdev= 8.98
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15],
| 30.00th=[ 21], 40.00th=[ 22], 50.00th=[ 30], 60.00th=[ 31],
| 70.00th=[ 31], 80.00th=[ 31], 90.00th=[ 31], 95.00th=[ 36],
| 99.00th=[ 53], 99.50th=[ 57], 99.90th=[ 83], 99.95th=[ 98],
| 99.99th=[ 115]
bw ( KiB/s): min=566520, max=570888, per=100.00%, avg=567735.86, stdev=1371.07, samples=14
iops : min=141630, max=142722, avg=141933.93, stdev=342.79, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=24.13%, 50=74.19%, 100=1.63%
lat (usec) : 250=0.04%
lat (msec) : 2=0.01%
cpu : usr=20.19%, sys=56.69%, ctx=253738, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=993445,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=554MiB/s (581MB/s), 554MiB/s-554MiB/s (581MB/s-581MB/s), io=3881MiB (4069MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-xnvme_1.txt"
$ cat /tmp/fio-output_4K_4_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734220,
"timestamp_ms" : 1666734220017,
"time" : "Tue Oct 25 23:43:40 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 4071936000,
"io_kbytes" : 3976500,
"bw_bytes" : 581622053,
"bw" : 567990,
"iops" : 141997.000429,
"runtime" : 7001,
"total_ios" : 994121,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2275,
"max" : 47789,
"mean" : 2511.122108,
"stddev" : 454.419258,
"N" : 994122
},
"clat_ns" : {
"min" : 271,
"max" : 165681,
"mean" : 25181.048494,
"stddev" : 8477.785485,
"N" : 994124,
"percentile" : {
"1.000000" : 14016,
"5.000000" : 14144,
"10.000000" : 14272,
"20.000000" : 14656,
"30.000000" : 20864,
"40.000000" : 21376,
"50.000000" : 29568,
"60.000000" : 30080,
"70.000000" : 30080,
"80.000000" : 30336,
"90.000000" : 30592,
"95.000000" : 34560,
"99.000000" : 52992,
"99.500000" : 56576,
"99.900000" : 81408,
"99.950000" : 96768,
"99.990000" : 115200
}
},
"lat_ns" : {
"min" : 11572,
"max" : 168056,
"mean" : 27692.170811,
"stddev" : 8578.284311,
"N" : 994124
},
"bw_min" : 565416,
"bw_max" : 571520,
"bw_agg" : 100.000000,
"bw_mean" : 568149.071429,
"bw_dev" : 1825.775977,
"bw_samples" : 14,
"iops_min" : 141354,
"iops_max" : 142880,
"iops_mean" : 142037.214286,
"iops_stddev" : 456.460661,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 18.328571,
"sys_cpu" : 58.714286,
"ctx" : 253915,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 24.695384,
"50" : 73.607136,
"100" : 1.651710,
"250" : 0.043355,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=88116: Tue Oct 25 23:43:40 2022
read: IOPS=142k, BW=555MiB/s (582MB/s)(3883MiB/7001msec)
slat (nsec): min=2275, max=47789, avg=2511.12, stdev=454.42
clat (nsec): min=271, max=165681, avg=25181.05, stdev=8477.79
lat (usec): min=11, max=168, avg=27.69, stdev= 8.58
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15],
| 30.00th=[ 21], 40.00th=[ 22], 50.00th=[ 30], 60.00th=[ 31],
| 70.00th=[ 31], 80.00th=[ 31], 90.00th=[ 31], 95.00th=[ 35],
| 99.00th=[ 53], 99.50th=[ 57], 99.90th=[ 82], 99.95th=[ 97],
| 99.99th=[ 116]
bw ( KiB/s): min=565416, max=571520, per=100.00%, avg=568149.07, stdev=1825.78, samples=14
iops : min=141354, max=142880, avg=142037.21, stdev=456.46, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=24.70%, 50=73.61%, 100=1.65%
lat (usec) : 250=0.04%
cpu : usr=18.33%, sys=58.71%, ctx=253915, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=994121,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=555MiB/s (582MB/s), 555MiB/s-555MiB/s (582MB/s-582MB/s), io=3883MiB (4072MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-xnvme_2.txt"
$ cat /tmp/fio-output_4K_4_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734233,
"timestamp_ms" : 1666734233669,
"time" : "Tue Oct 25 23:43:53 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 4061618176,
"io_kbytes" : 3966424,
"bw_bytes" : 580148289,
"bw" : 566551,
"iops" : 141637.194686,
"runtime" : 7001,
"total_ios" : 991602,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2294,
"max" : 197260,
"mean" : 2527.106294,
"stddev" : 495.502687,
"N" : 991603
},
"clat_ns" : {
"min" : 230,
"max" : 212398,
"mean" : 25205.674751,
"stddev" : 8552.840582,
"N" : 991605,
"percentile" : {
"1.000000" : 14016,
"5.000000" : 14144,
"10.000000" : 14272,
"20.000000" : 14528,
"30.000000" : 20864,
"40.000000" : 21120,
"50.000000" : 29824,
"60.000000" : 30080,
"70.000000" : 30336,
"80.000000" : 30336,
"90.000000" : 30592,
"95.000000" : 35072,
"99.000000" : 52480,
"99.500000" : 55552,
"99.900000" : 84480,
"99.950000" : 98816,
"99.990000" : 116224
}
},
"lat_ns" : {
"min" : 11661,
"max" : 215053,
"mean" : 27732.780705,
"stddev" : 8653.005043,
"N" : 991605
},
"bw_min" : 565176,
"bw_max" : 569450,
"bw_agg" : 100.000000,
"bw_mean" : 566683.571429,
"bw_dev" : 1091.546871,
"bw_samples" : 14,
"iops_min" : 141294,
"iops_max" : 142362,
"iops_mean" : 141670.857143,
"iops_stddev" : 272.789256,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 21.257143,
"sys_cpu" : 55.942857,
"ctx" : 253051,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 23.960924,
"50" : 74.303501,
"100" : 1.684345,
"250" : 0.046793,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=88156: Tue Oct 25 23:43:53 2022
read: IOPS=142k, BW=553MiB/s (580MB/s)(3873MiB/7001msec)
slat (nsec): min=2294, max=197260, avg=2527.11, stdev=495.50
clat (nsec): min=230, max=212398, avg=25205.67, stdev=8552.84
lat (usec): min=11, max=215, avg=27.73, stdev= 8.65
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15],
| 30.00th=[ 21], 40.00th=[ 22], 50.00th=[ 30], 60.00th=[ 31],
| 70.00th=[ 31], 80.00th=[ 31], 90.00th=[ 31], 95.00th=[ 36],
| 99.00th=[ 53], 99.50th=[ 56], 99.90th=[ 85], 99.95th=[ 99],
| 99.99th=[ 117]
bw ( KiB/s): min=565176, max=569450, per=100.00%, avg=566683.57, stdev=1091.55, samples=14
iops : min=141294, max=142362, avg=141670.86, stdev=272.79, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=23.96%, 50=74.30%, 100=1.68%
lat (usec) : 250=0.05%
cpu : usr=21.26%, sys=55.94%, ctx=253051, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=991602,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=553MiB/s (580MB/s), 553MiB/s-553MiB/s (580MB/s-580MB/s), io=3873MiB (4062MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-reference_0.txt"
$ cat /tmp/fio-output_4K_4_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734247,
"timestamp_ms" : 1666734247308,
"time" : "Tue Oct 25 23:44:07 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 5208461312,
"io_kbytes" : 5086388,
"bw_bytes" : 743959621,
"bw" : 726523,
"iops" : 181630.195686,
"runtime" : 7001,
"total_ios" : 1271593,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1994,
"max" : 27552,
"mean" : 2273.459399,
"stddev" : 184.930489,
"N" : 1271594
},
"clat_ns" : {
"min" : 6773,
"max" : 133701,
"mean" : 19356.888477,
"stddev" : 6100.168271,
"N" : 1271596,
"percentile" : {
"1.000000" : 14400,
"5.000000" : 14528,
"10.000000" : 14656,
"20.000000" : 14784,
"30.000000" : 15168,
"40.000000" : 15424,
"50.000000" : 18560,
"60.000000" : 21120,
"70.000000" : 21120,
"80.000000" : 21376,
"90.000000" : 21888,
"95.000000" : 30848,
"99.000000" : 42240,
"99.500000" : 47360,
"99.900000" : 77312,
"99.950000" : 91648,
"99.990000" : 104960
}
},
"lat_ns" : {
"min" : 11512,
"max" : 135855,
"mean" : 21630.347846,
"stddev" : 6127.066883,
"N" : 1271596
},
"bw_min" : 724216,
"bw_max" : 728976,
"bw_agg" : 100.000000,
"bw_mean" : 726728.285714,
"bw_dev" : 1331.121877,
"bw_samples" : 14,
"iops_min" : 181054,
"iops_max" : 182244,
"iops_mean" : 181682.071429,
"iops_stddev" : 332.780469,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 21.585714,
"sys_cpu" : 48.000000,
"ctx" : 330200,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 50.625082,
"50" : 49.125703,
"100" : 0.219253,
"250" : 0.023750,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 1806255,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 26402,
"write_ticks" : 0,
"in_queue" : 26402,
"util" : 99.078934
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=88195: Tue Oct 25 23:44:07 2022
read: IOPS=182k, BW=709MiB/s (744MB/s)(4967MiB/7001msec)
slat (nsec): min=1994, max=27552, avg=2273.46, stdev=184.93
clat (usec): min=6, max=133, avg=19.36, stdev= 6.10
lat (usec): min=11, max=135, avg=21.63, stdev= 6.13
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15],
| 30.00th=[ 16], 40.00th=[ 16], 50.00th=[ 19], 60.00th=[ 22],
| 70.00th=[ 22], 80.00th=[ 22], 90.00th=[ 22], 95.00th=[ 31],
| 99.00th=[ 43], 99.50th=[ 48], 99.90th=[ 78], 99.95th=[ 92],
| 99.99th=[ 105]
bw ( KiB/s): min=724216, max=728976, per=100.00%, avg=726728.29, stdev=1331.12, samples=14
iops : min=181054, max=182244, avg=181682.07, stdev=332.78, samples=14
lat (usec) : 10=0.01%, 20=50.63%, 50=49.13%, 100=0.22%, 250=0.02%
cpu : usr=21.59%, sys=48.00%, ctx=330200, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1271593,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=709MiB/s (744MB/s), 709MiB/s-709MiB/s (744MB/s-744MB/s), io=4967MiB (5208MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=1806255/0, merge=0/0, ticks=26402/0, in_queue=26402, util=99.08%
$ rm /tmp/fio-output_4K_4_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-reference_1.txt"
$ cat /tmp/fio-output_4K_4_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734260,
"timestamp_ms" : 1666734260967,
"time" : "Tue Oct 25 23:44:20 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 5129441280,
"io_kbytes" : 5009220,
"bw_bytes" : 732672658,
"bw" : 715500,
"iops" : 178874.589344,
"runtime" : 7001,
"total_ios" : 1252301,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1342,
"max" : 38192,
"mean" : 2214.863789,
"stddev" : 1428.389632,
"N" : 1252302
},
"clat_ns" : {
"min" : 2564,
"max" : 191189,
"mean" : 19851.901731,
"stddev" : 5831.909239,
"N" : 1252304,
"percentile" : {
"1.000000" : 14016,
"5.000000" : 15040,
"10.000000" : 15552,
"20.000000" : 15808,
"30.000000" : 17280,
"40.000000" : 17536,
"50.000000" : 19072,
"60.000000" : 19584,
"70.000000" : 21376,
"80.000000" : 21376,
"90.000000" : 21888,
"95.000000" : 31616,
"99.000000" : 43264,
"99.500000" : 46336,
"99.900000" : 78336,
"99.950000" : 92672,
"99.990000" : 104960
}
},
"lat_ns" : {
"min" : 10289,
"max" : 195036,
"mean" : 22066.766055,
"stddev" : 5810.190816,
"N" : 1252304
},
"bw_min" : 705904,
"bw_max" : 720104,
"bw_agg" : 100.000000,
"bw_mean" : 715783.785714,
"bw_dev" : 3968.759430,
"bw_samples" : 14,
"iops_min" : 176476,
"iops_max" : 180026,
"iops_mean" : 178945.857143,
"iops_stddev" : 992.094820,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.257143,
"sys_cpu" : 45.128571,
"ctx" : 319706,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.063084,
"20" : 61.387717,
"50" : 38.299898,
"100" : 0.221991,
"250" : 0.026990,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 1786396,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 28771,
"write_ticks" : 0,
"in_queue" : 28771,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=88233: Tue Oct 25 23:44:20 2022
read: IOPS=179k, BW=699MiB/s (733MB/s)(4892MiB/7001msec)
slat (nsec): min=1342, max=38192, avg=2214.86, stdev=1428.39
clat (usec): min=2, max=191, avg=19.85, stdev= 5.83
lat (usec): min=10, max=195, avg=22.07, stdev= 5.81
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16],
| 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 20],
| 70.00th=[ 22], 80.00th=[ 22], 90.00th=[ 22], 95.00th=[ 32],
| 99.00th=[ 44], 99.50th=[ 47], 99.90th=[ 79], 99.95th=[ 93],
| 99.99th=[ 105]
bw ( KiB/s): min=705904, max=720104, per=100.00%, avg=715783.79, stdev=3968.76, samples=14
iops : min=176476, max=180026, avg=178945.86, stdev=992.09, samples=14
lat (usec) : 4=0.01%, 10=0.06%, 20=61.39%, 50=38.30%, 100=0.22%
lat (usec) : 250=0.03%
cpu : usr=17.26%, sys=45.13%, ctx=319706, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1252301,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=699MiB/s (733MB/s), 699MiB/s-699MiB/s (733MB/s-733MB/s), io=4892MiB (5129MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=1786396/0, merge=0/0, ticks=28771/0, in_queue=28771, util=99.09%
$ rm /tmp/fio-output_4K_4_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-reference_2.txt"
$ cat /tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733116,
"timestamp_ms" : 1666733116091,
"time" : "Tue Oct 25 23:25:16 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2407424000,
"io_kbytes" : 2351000,
"bw_bytes" : 343868590,
"bw" : 335809,
"iops" : 83952.292530,
"runtime" : 7001,
"total_ios" : 587750,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 711,
"max" : 11492,
"mean" : 925.037659,
"stddev" : 79.223115,
"N" : 587750
},
"clat_ns" : {
"min" : 3837,
"max" : 176060,
"mean" : 10698.287205,
"stddev" : 2205.678474,
"N" : 587750,
"percentile" : {
"1.000000" : 9920,
"5.000000" : 10048,
"10.000000" : 10048,
"20.000000" : 10176,
"30.000000" : 10304,
"40.000000" : 10304,
"50.000000" : 10432,
"60.000000" : 10432,
"70.000000" : 10560,
"80.000000" : 10560,
"90.000000" : 10688,
"95.000000" : 11072,
"99.000000" : 19840,
"99.500000" : 27520,
"99.900000" : 28032,
"99.950000" : 43776,
"99.990000" : 67072
}
},
"lat_ns" : {
"min" : 10570,
"max" : 179687,
"mean" : 11623.324864,
"stddev" : 2208.087793,
"N" : 587750
},
"bw_min" : 335496,
"bw_max" : 336312,
"bw_agg" : 100.000000,
"bw_mean" : 335897.642857,
"bw_dev" : 239.573213,
"bw_samples" : 14,
"iops_min" : 83874,
"iops_max" : 84078,
"iops_mean" : 83974.357143,
"iops_stddev" : 59.925091,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 20,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 3.303956,
"20" : 95.712123,
"50" : 0.938494,
"100" : 0.044747,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=84733: Tue Oct 25 23:25:16 2022
read: IOPS=84.0k, BW=328MiB/s (344MB/s)(2296MiB/7001msec)
slat (nsec): min=711, max=11492, avg=925.04, stdev=79.22
clat (usec): min=3, max=176, avg=10.70, stdev= 2.21
lat (usec): min=10, max=179, avg=11.62, stdev= 2.21
clat percentiles (nsec):
| 1.00th=[ 9920], 5.00th=[10048], 10.00th=[10048], 20.00th=[10176],
| 30.00th=[10304], 40.00th=[10304], 50.00th=[10432], 60.00th=[10432],
| 70.00th=[10560], 80.00th=[10560], 90.00th=[10688], 95.00th=[11072],
| 99.00th=[19840], 99.50th=[27520], 99.90th=[28032], 99.95th=[43776],
| 99.99th=[67072]
bw ( KiB/s): min=335496, max=336312, per=100.00%, avg=335897.64, stdev=239.57, samples=14
iops : min=83874, max=84078, avg=83974.36, stdev=59.93, samples=14
lat (usec) : 4=0.01%, 10=3.30%, 20=95.71%, 50=0.94%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=20, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=587750,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=2296MiB (2407MB), run=7001-7001msec
$ cat /tmp/fio-output_4K_4_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734274,
"timestamp_ms" : 1666734274606,
"time" : "Tue Oct 25 23:44:34 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 5094621184,
"io_kbytes" : 4975216,
"bw_bytes" : 727699069,
"bw" : 710643,
"iops" : 177660.477075,
"runtime" : 7001,
"total_ios" : 1243801,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1322,
"max" : 179357,
"mean" : 2270.165353,
"stddev" : 1482.663537,
"N" : 1243801
},
"clat_ns" : {
"min" : 2415,
"max" : 1314775,
"mean" : 19921.951998,
"stddev" : 6421.548737,
"N" : 1243804,
"percentile" : {
"1.000000" : 14016,
"5.000000" : 15040,
"10.000000" : 15552,
"20.000000" : 15808,
"30.000000" : 17280,
"40.000000" : 17536,
"50.000000" : 19072,
"60.000000" : 19328,
"70.000000" : 21120,
"80.000000" : 21376,
"90.000000" : 23680,
"95.000000" : 31872,
"99.000000" : 43264,
"99.500000" : 47360,
"99.900000" : 79360,
"99.950000" : 93696,
"99.990000" : 104960
}
},
"lat_ns" : {
"min" : 9989,
"max" : 1316138,
"mean" : 22192.118014,
"stddev" : 6367.348917,
"N" : 1243804
},
"bw_min" : 653496,
"bw_max" : 722680,
"bw_agg" : 100.000000,
"bw_mean" : 710909.000000,
"bw_dev" : 18576.458195,
"bw_samples" : 14,
"iops_min" : 163374,
"iops_max" : 180670,
"iops_mean" : 177727.285714,
"iops_stddev" : 4644.186298,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.771429,
"sys_cpu" : 45.885714,
"ctx" : 313922,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.049365,
"20" : 62.858367,
"50" : 36.778311,
"100" : 0.285978,
"250" : 0.027336,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 1774792,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 28485,
"write_ticks" : 0,
"in_queue" : 28485,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=88271: Tue Oct 25 23:44:34 2022
read: IOPS=178k, BW=694MiB/s (728MB/s)(4859MiB/7001msec)
slat (nsec): min=1322, max=179357, avg=2270.17, stdev=1482.66
clat (usec): min=2, max=1314, avg=19.92, stdev= 6.42
lat (usec): min=9, max=1316, avg=22.19, stdev= 6.37
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16],
| 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 20],
| 70.00th=[ 22], 80.00th=[ 22], 90.00th=[ 24], 95.00th=[ 32],
| 99.00th=[ 44], 99.50th=[ 48], 99.90th=[ 80], 99.95th=[ 94],
| 99.99th=[ 105]
bw ( KiB/s): min=653496, max=722680, per=100.00%, avg=710909.00, stdev=18576.46, samples=14
iops : min=163374, max=180670, avg=177727.29, stdev=4644.19, samples=14
lat (usec) : 4=0.01%, 10=0.05%, 20=62.86%, 50=36.78%, 100=0.29%
lat (usec) : 250=0.03%
lat (msec) : 2=0.01%
cpu : usr=17.77%, sys=45.89%, ctx=313922, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1243801,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=694MiB/s (728MB/s), 694MiB/s-694MiB/s (728MB/s-728MB/s), io=4859MiB (5095MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=1774792/0, merge=0/0, ticks=28485/0, in_queue=28485, util=99.09%
$ rm /tmp/fio-output_4K_4_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-bdev_aio_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-bdev_aio_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_4_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734289,
"timestamp_ms" : 1666734289535,
"time" : "Tue Oct 25 23:44:49 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6368309248,
"io_kbytes" : 6219052,
"bw_bytes" : 909628517,
"bw" : 888309,
"iops" : 222076.846165,
"runtime" : 7001,
"total_ios" : 1554760,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1523,
"max" : 20198,
"mean" : 1636.536629,
"stddev" : 223.818184,
"N" : 1554760
},
"clat_ns" : {
"min" : 251,
"max" : 148058,
"mean" : 16084.288581,
"stddev" : 5476.140778,
"N" : 1554763,
"percentile" : {
"1.000000" : 13504,
"5.000000" : 14400,
"10.000000" : 14528,
"20.000000" : 14528,
"30.000000" : 14656,
"40.000000" : 14656,
"50.000000" : 14656,
"60.000000" : 14784,
"70.000000" : 14784,
"80.000000" : 14912,
"90.000000" : 16512,
"95.000000" : 27008,
"99.000000" : 36096,
"99.500000" : 42752,
"99.900000" : 79360,
"99.950000" : 90624,
"99.990000" : 104960
}
},
"lat_ns" : {
"min" : 9618,
"max" : 149671,
"mean" : 17720.825126,
"stddev" : 5474.918573,
"N" : 1554763
},
"bw_min" : 886712,
"bw_max" : 890216,
"bw_agg" : 100.000000,
"bw_mean" : 888525.785714,
"bw_dev" : 1043.348321,
"bw_samples" : 14,
"iops_min" : 221678,
"iops_max" : 222554,
"iops_mean" : 222131.428571,
"iops_stddev" : 260.832311,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 73.700000,
"sys_cpu" : 26.285714,
"ctx" : 4,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.372984,
"2" : 0.960470,
"4" : 98.666547,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.063933,
"20" : 92.064048,
"50" : 7.686588,
"100" : 0.157516,
"250" : 0.025663,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=88319: Tue Oct 25 23:44:49 2022
read: IOPS=222k, BW=867MiB/s (910MB/s)(6073MiB/7001msec)
slat (nsec): min=1523, max=20198, avg=1636.54, stdev=223.82
clat (nsec): min=251, max=148058, avg=16084.29, stdev=5476.14
lat (usec): min=9, max=149, avg=17.72, stdev= 5.47
clat percentiles (usec):
| 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15],
| 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 15],
| 70.00th=[ 15], 80.00th=[ 15], 90.00th=[ 17], 95.00th=[ 28],
| 99.00th=[ 37], 99.50th=[ 43], 99.90th=[ 80], 99.95th=[ 91],
| 99.99th=[ 105]
bw ( KiB/s): min=886712, max=890216, per=100.00%, avg=888525.79, stdev=1043.35, samples=14
iops : min=221678, max=222554, avg=222131.43, stdev=260.83, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.06%, 20=92.06%, 50=7.69%, 100=0.16%
lat (usec) : 250=0.03%
cpu : usr=73.70%, sys=26.29%, ctx=4, majf=0, minf=0
IO depths : 1=0.4%, 2=1.0%, 4=98.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1554760,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=867MiB/s (910MB/s), 867MiB/s-867MiB/s (910MB/s-910MB/s), io=6073MiB (6368MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-bdev_aio_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-bdev_aio_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_4_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734304,
"timestamp_ms" : 1666734304702,
"time" : "Tue Oct 25 23:45:04 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6271488000,
"io_kbytes" : 6124500,
"bw_bytes" : 895798885,
"bw" : 874803,
"iops" : 218700.328524,
"runtime" : 7001,
"total_ios" : 1531121,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1643,
"max" : 62267,
"mean" : 1883.836522,
"stddev" : 167.538851,
"N" : 1531122
},
"clat_ns" : {
"min" : 290,
"max" : 141535,
"mean" : 16162.363481,
"stddev" : 6497.223878,
"N" : 1531121,
"percentile" : {
"1.000000" : 10432,
"5.000000" : 10432,
"10.000000" : 10560,
"20.000000" : 12480,
"30.000000" : 12864,
"40.000000" : 14272,
"50.000000" : 14528,
"60.000000" : 14784,
"70.000000" : 17024,
"80.000000" : 19328,
"90.000000" : 21376,
"95.000000" : 28800,
"99.000000" : 41216,
"99.500000" : 44800,
"99.900000" : 77312,
"99.950000" : 94720,
"99.990000" : 121344
}
},
"lat_ns" : {
"min" : 9808,
"max" : 143308,
"mean" : 18046.199945,
"stddev" : 6499.210762,
"N" : 1531121
},
"bw_min" : 873272,
"bw_max" : 877680,
"bw_agg" : 100.000000,
"bw_mean" : 875229.571429,
"bw_dev" : 1466.632654,
"bw_samples" : 14,
"iops_min" : 218318,
"iops_max" : 219420,
"iops_mean" : 218807.285714,
"iops_stddev" : 366.406758,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 77.814286,
"sys_cpu" : 22.157143,
"ctx" : 22,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.441180,
"4" : 99.548239,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.063418,
"20" : 80.184910,
"50" : 19.526347,
"100" : 0.182611,
"250" : 0.041016,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=88368: Tue Oct 25 23:45:04 2022
read: IOPS=219k, BW=854MiB/s (896MB/s)(5981MiB/7001msec)
slat (nsec): min=1643, max=62267, avg=1883.84, stdev=167.54
clat (nsec): min=290, max=141535, avg=16162.36, stdev=6497.22
lat (usec): min=9, max=143, avg=18.05, stdev= 6.50
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 13],
| 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 15],
| 70.00th=[ 18], 80.00th=[ 20], 90.00th=[ 22], 95.00th=[ 29],
| 99.00th=[ 42], 99.50th=[ 45], 99.90th=[ 78], 99.95th=[ 95],
| 99.99th=[ 122]
bw ( KiB/s): min=873272, max=877680, per=100.00%, avg=875229.57, stdev=1466.63, samples=14
iops : min=218318, max=219420, avg=218807.29, stdev=366.41, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 10=0.06%, 20=80.18%, 50=19.53%, 100=0.18%, 250=0.04%
cpu : usr=77.81%, sys=22.16%, ctx=22, majf=0, minf=0
IO depths : 1=0.1%, 2=0.4%, 4=99.5%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1531121,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=854MiB/s (896MB/s), 854MiB/s-854MiB/s (896MB/s-896MB/s), io=5981MiB (6271MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-bdev_aio_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-bdev_aio_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_4_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734319,
"timestamp_ms" : 1666734319976,
"time" : "Tue Oct 25 23:45:19 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6364598272,
"io_kbytes" : 6215428,
"bw_bytes" : 909098453,
"bw" : 887791,
"iops" : 221947.293244,
"runtime" : 7001,
"total_ios" : 1553853,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1513,
"max" : 25969,
"mean" : 1631.607934,
"stddev" : 243.107289,
"N" : 1553854
},
"clat_ns" : {
"min" : 261,
"max" : 131006,
"mean" : 16099.061237,
"stddev" : 5450.853875,
"N" : 1553853,
"percentile" : {
"1.000000" : 13504,
"5.000000" : 14400,
"10.000000" : 14528,
"20.000000" : 14528,
"30.000000" : 14656,
"40.000000" : 14656,
"50.000000" : 14656,
"60.000000" : 14784,
"70.000000" : 14784,
"80.000000" : 14912,
"90.000000" : 16512,
"95.000000" : 27008,
"99.000000" : 36096,
"99.500000" : 42752,
"99.900000" : 79360,
"99.950000" : 88576,
"99.990000" : 103936
}
},
"lat_ns" : {
"min" : 9708,
"max" : 132579,
"mean" : 17730.669197,
"stddev" : 5450.382413,
"N" : 1553853
},
"bw_min" : 886560,
"bw_max" : 889792,
"bw_agg" : 100.000000,
"bw_mean" : 888029.857143,
"bw_dev" : 1001.853195,
"bw_samples" : 14,
"iops_min" : 221640,
"iops_max" : 222448,
"iops_mean" : 222007.428571,
"iops_stddev" : 250.424168,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 69.928571,
"sys_cpu" : 30.042857,
"ctx" : 4,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.372300,
"2" : 0.969268,
"4" : 98.658432,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.058757,
"20" : 91.962818,
"50" : 7.797906,
"100" : 0.153618,
"250" : 0.024327,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=88418: Tue Oct 25 23:45:19 2022
read: IOPS=222k, BW=867MiB/s (909MB/s)(6070MiB/7001msec)
slat (nsec): min=1513, max=25969, avg=1631.61, stdev=243.11
clat (nsec): min=261, max=131006, avg=16099.06, stdev=5450.85
lat (usec): min=9, max=132, avg=17.73, stdev= 5.45
clat percentiles (usec):
| 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15],
| 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 15],
| 70.00th=[ 15], 80.00th=[ 15], 90.00th=[ 17], 95.00th=[ 28],
| 99.00th=[ 37], 99.50th=[ 43], 99.90th=[ 80], 99.95th=[ 89],
| 99.99th=[ 104]
bw ( KiB/s): min=886560, max=889792, per=100.00%, avg=888029.86, stdev=1001.85, samples=14
iops : min=221640, max=222448, avg=222007.43, stdev=250.42, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 10=0.06%, 20=91.96%, 50=7.80%, 100=0.15%, 250=0.02%
cpu : usr=69.93%, sys=30.04%, ctx=4, majf=0, minf=0
IO depths : 1=0.4%, 2=1.0%, 4=98.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1553853,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=867MiB/s (909MB/s), 867MiB/s-867MiB/s (909MB/s-909MB/s), io=6070MiB (6365MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-reference_0.txt': No such file or directory
$ rm /tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734335,
"timestamp_ms" : 1666734335155,
"time" : "Tue Oct 25 23:45:35 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6271553536,
"io_kbytes" : 6124564,
"bw_bytes" : 895808246,
"bw" : 874812,
"iops" : 218702.756749,
"runtime" : 7001,
"total_ios" : 1531138,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1552,
"max" : 172423,
"mean" : 2025.800648,
"stddev" : 791.208125,
"N" : 1531138
},
"clat_ns" : {
"min" : 651,
"max" : 183304,
"mean" : 15858.742410,
"stddev" : 5803.263051,
"N" : 1531141,
"percentile" : {
"1.000000" : 10176,
"5.000000" : 10560,
"10.000000" : 12992,
"20.000000" : 13760,
"30.000000" : 14016,
"40.000000" : 14272,
"50.000000" : 14528,
"60.000000" : 14784,
"70.000000" : 15168,
"80.000000" : 16192,
"90.000000" : 17792,
"95.000000" : 27520,
"99.000000" : 38656,
"99.500000" : 42752,
"99.900000" : 79360,
"99.950000" : 90624,
"99.990000" : 103936
}
},
"lat_ns" : {
"min" : 9658,
"max" : 186690,
"mean" : 17884.542243,
"stddev" : 5775.342587,
"N" : 1531141
},
"bw_min" : 872848,
"bw_max" : 876888,
"bw_agg" : 100.000000,
"bw_mean" : 875033.785714,
"bw_dev" : 1002.540326,
"bw_samples" : 14,
"iops_min" : 218212,
"iops_max" : 219222,
"iops_mean" : 218758.428571,
"iops_stddev" : 250.748024,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 42.671429,
"sys_cpu" : 57.300000,
"ctx" : 12,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.273522,
"2" : 6.079334,
"4" : 93.647143,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.020834,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.145970,
"20" : 92.134086,
"50" : 7.513366,
"100" : 0.155897,
"250" : 0.024753,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=88468: Tue Oct 25 23:45:35 2022
read: IOPS=219k, BW=854MiB/s (896MB/s)(5981MiB/7001msec)
slat (nsec): min=1552, max=172423, avg=2025.80, stdev=791.21
clat (nsec): min=651, max=183304, avg=15858.74, stdev=5803.26
lat (usec): min=9, max=186, avg=17.88, stdev= 5.78
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 14],
| 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 15],
| 70.00th=[ 16], 80.00th=[ 17], 90.00th=[ 18], 95.00th=[ 28],
| 99.00th=[ 39], 99.50th=[ 43], 99.90th=[ 80], 99.95th=[ 91],
| 99.99th=[ 104]
bw ( KiB/s): min=872848, max=876888, per=100.00%, avg=875033.79, stdev=1002.54, samples=14
iops : min=218212, max=219222, avg=218758.43, stdev=250.75, samples=14
lat (nsec) : 750=0.02%, 1000=0.01%
lat (usec) : 4=0.01%, 10=0.15%, 20=92.13%, 50=7.51%, 100=0.16%
lat (usec) : 250=0.02%
cpu : usr=42.67%, sys=57.30%, ctx=12, majf=0, minf=0
IO depths : 1=0.3%, 2=6.1%, 4=93.6%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1531138,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=854MiB/s (896MB/s), 854MiB/s-854MiB/s (896MB/s-896MB/s), io=5981MiB (6272MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734350,
"timestamp_ms" : 1666734350304,
"time" : "Tue Oct 25 23:45:50 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6267457536,
"io_kbytes" : 6120564,
"bw_bytes" : 895223187,
"bw" : 874241,
"iops" : 218559.777175,
"runtime" : 7001,
"total_ios" : 1530137,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1532,
"max" : 20007,
"mean" : 2014.375794,
"stddev" : 757.866366,
"N" : 1530138
},
"clat_ns" : {
"min" : 641,
"max" : 144321,
"mean" : 15884.766559,
"stddev" : 5830.698008,
"N" : 1530137,
"percentile" : {
"1.000000" : 10176,
"5.000000" : 10560,
"10.000000" : 12992,
"20.000000" : 13760,
"30.000000" : 14016,
"40.000000" : 14272,
"50.000000" : 14528,
"60.000000" : 14784,
"70.000000" : 15168,
"80.000000" : 16192,
"90.000000" : 18048,
"95.000000" : 27520,
"99.000000" : 38656,
"99.500000" : 42752,
"99.900000" : 80384,
"99.950000" : 90624,
"99.990000" : 103936
}
},
"lat_ns" : {
"min" : 9969,
"max" : 145914,
"mean" : 17899.142622,
"stddev" : 5795.903965,
"N" : 1530137
},
"bw_min" : 871464,
"bw_max" : 876000,
"bw_agg" : 100.000000,
"bw_mean" : 874451.071429,
"bw_dev" : 1455.933772,
"bw_samples" : 14,
"iops_min" : 217866,
"iops_max" : 219000,
"iops_mean" : 218612.857143,
"iops_stddev" : 364.172161,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 41.014286,
"sys_cpu" : 58.957143,
"ctx" : 6,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.280106,
"2" : 6.138274,
"4" : 93.581620,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.018953,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.173122,
"20" : 92.036465,
"50" : 7.573636,
"100" : 0.166521,
"250" : 0.026403,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=88518: Tue Oct 25 23:45:50 2022
read: IOPS=219k, BW=854MiB/s (895MB/s)(5977MiB/7001msec)
slat (nsec): min=1532, max=20007, avg=2014.38, stdev=757.87
clat (nsec): min=641, max=144321, avg=15884.77, stdev=5830.70
lat (usec): min=9, max=145, avg=17.90, stdev= 5.80
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 14],
| 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 15],
| 70.00th=[ 16], 80.00th=[ 17], 90.00th=[ 19], 95.00th=[ 28],
| 99.00th=[ 39], 99.50th=[ 43], 99.90th=[ 81], 99.95th=[ 91],
| 99.99th=[ 104]
bw ( KiB/s): min=871464, max=876000, per=100.00%, avg=874451.07, stdev=1455.93, samples=14
iops : min=217866, max=219000, avg=218612.86, stdev=364.17, samples=14
lat (nsec) : 750=0.02%, 1000=0.01%
lat (usec) : 4=0.01%, 10=0.17%, 20=92.04%, 50=7.57%, 100=0.17%
lat (usec) : 250=0.03%
cpu : usr=41.01%, sys=58.96%, ctx=6, majf=0, minf=0
IO depths : 1=0.3%, 2=6.1%, 4=93.6%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1530137,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=854MiB/s (895MB/s), 854MiB/s-854MiB/s (895MB/s-895MB/s), io=5977MiB (6267MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734365,
"timestamp_ms" : 1666734365443,
"time" : "Tue Oct 25 23:46:05 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6263152640,
"io_kbytes" : 6116360,
"bw_bytes" : 894608290,
"bw" : 873640,
"iops" : 218409.798600,
"runtime" : 7001,
"total_ios" : 1529087,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1552,
"max" : 1296060,
"mean" : 2038.297024,
"stddev" : 1293.969014,
"N" : 1529087
},
"clat_ns" : {
"min" : 652,
"max" : 1309395,
"mean" : 15867.342854,
"stddev" : 6045.816309,
"N" : 1529090,
"percentile" : {
"1.000000" : 10304,
"5.000000" : 10560,
"10.000000" : 12992,
"20.000000" : 13760,
"30.000000" : 14016,
"40.000000" : 14400,
"50.000000" : 14656,
"60.000000" : 14784,
"70.000000" : 15168,
"80.000000" : 16192,
"90.000000" : 18048,
"95.000000" : 27520,
"99.000000" : 38656,
"99.500000" : 42752,
"99.900000" : 78336,
"99.950000" : 90624,
"99.990000" : 103936
}
},
"lat_ns" : {
"min" : 9738,
"max" : 1311068,
"mean" : 17905.639056,
"stddev" : 6101.138943,
"N" : 1529090
},
"bw_min" : 871208,
"bw_max" : 876849,
"bw_agg" : 100.000000,
"bw_mean" : 873847.500000,
"bw_dev" : 1583.292322,
"bw_samples" : 14,
"iops_min" : 217802,
"iops_max" : 219212,
"iops_mean" : 218461.857143,
"iops_stddev" : 395.786628,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 41.414286,
"sys_cpu" : 58.557143,
"ctx" : 45,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.279579,
"2" : 6.606622,
"4" : 93.113799,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.020470,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.132170,
"20" : 92.125039,
"50" : 7.538289,
"100" : 0.154994,
"250" : 0.023413,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=88567: Tue Oct 25 23:46:05 2022
read: IOPS=218k, BW=853MiB/s (895MB/s)(5973MiB/7001msec)
slat (nsec): min=1552, max=1296.1k, avg=2038.30, stdev=1293.97
clat (nsec): min=652, max=1309.4k, avg=15867.34, stdev=6045.82
lat (usec): min=9, max=1311, avg=17.91, stdev= 6.10
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 14],
| 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 15],
| 70.00th=[ 16], 80.00th=[ 17], 90.00th=[ 19], 95.00th=[ 28],
| 99.00th=[ 39], 99.50th=[ 43], 99.90th=[ 79], 99.95th=[ 91],
| 99.99th=[ 104]
bw ( KiB/s): min=871208, max=876849, per=100.00%, avg=873847.50, stdev=1583.29, samples=14
iops : min=217802, max=219212, avg=218461.86, stdev=395.79, samples=14
lat (nsec) : 750=0.02%, 1000=0.01%
lat (usec) : 4=0.01%, 10=0.13%, 20=92.13%, 50=7.54%, 100=0.15%
lat (usec) : 250=0.02%
lat (msec) : 2=0.01%
cpu : usr=41.41%, sys=58.56%, ctx=45, majf=0, minf=0
IO depths : 1=0.3%, 2=6.6%, 4=93.1%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1529087,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=853MiB/s (895MB/s), 853MiB/s-853MiB/s (895MB/s-895MB/s), io=5973MiB (6263MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-reference_0.txt"
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-xnvme_0.txt"
$ cat /tmp/fio-output_4K_4_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734379,
"timestamp_ms" : 1666734379201,
"time" : "Tue Oct 25 23:46:19 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 5072072704,
"io_kbytes" : 4953196,
"bw_bytes" : 724478317,
"bw" : 707498,
"iops" : 176874.160834,
"runtime" : 7001,
"total_ios" : 1238296,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1392,
"max" : 24916,
"mean" : 2289.736705,
"stddev" : 1489.650647,
"N" : 1238296
},
"clat_ns" : {
"min" : 2695,
"max" : 133280,
"mean" : 20010.147292,
"stddev" : 5825.442065,
"N" : 1238299,
"percentile" : {
"1.000000" : 14912,
"5.000000" : 15680,
"10.000000" : 15808,
"20.000000" : 16064,
"30.000000" : 17280,
"40.000000" : 17536,
"50.000000" : 19328,
"60.000000" : 19584,
"70.000000" : 21376,
"80.000000" : 21888,
"90.000000" : 22144,
"95.000000" : 31360,
"99.000000" : 43776,
"99.500000" : 46848,
"99.900000" : 81408,
"99.950000" : 93696,
"99.990000" : 105984
}
},
"lat_ns" : {
"min" : 10369,
"max" : 134742,
"mean" : 22299.882204,
"stddev" : 5670.530977,
"N" : 1238299
},
"bw_min" : 703320,
"bw_max" : 710557,
"bw_agg" : 100.000000,
"bw_mean" : 707687.785714,
"bw_dev" : 1699.738812,
"bw_samples" : 14,
"iops_min" : 175830,
"iops_max" : 177639,
"iops_mean" : 176921.928571,
"iops_stddev" : 424.902245,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.414286,
"sys_cpu" : 50.042857,
"ctx" : 319314,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.027215,
"20" : 65.634549,
"50" : 34.085227,
"100" : 0.224744,
"250" : 0.028345,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=88606: Tue Oct 25 23:46:19 2022
read: IOPS=177k, BW=691MiB/s (724MB/s)(4837MiB/7001msec)
slat (nsec): min=1392, max=24916, avg=2289.74, stdev=1489.65
clat (usec): min=2, max=133, avg=20.01, stdev= 5.83
lat (usec): min=10, max=134, avg=22.30, stdev= 5.67
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17],
| 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 20],
| 70.00th=[ 22], 80.00th=[ 22], 90.00th=[ 23], 95.00th=[ 32],
| 99.00th=[ 44], 99.50th=[ 47], 99.90th=[ 82], 99.95th=[ 94],
| 99.99th=[ 106]
bw ( KiB/s): min=703320, max=710557, per=100.00%, avg=707687.79, stdev=1699.74, samples=14
iops : min=175830, max=177639, avg=176921.93, stdev=424.90, samples=14
lat (usec) : 4=0.01%, 10=0.03%, 20=65.63%, 50=34.09%, 100=0.22%
lat (usec) : 250=0.03%
cpu : usr=15.41%, sys=50.04%, ctx=319314, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1238296,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=691MiB/s (724MB/s), 691MiB/s-691MiB/s (724MB/s-724MB/s), io=4837MiB (5072MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-xnvme_1.txt"
$ cat /tmp/fio-output_4K_4_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734392,
"timestamp_ms" : 1666734392857,
"time" : "Tue Oct 25 23:46:32 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 5079027712,
"io_kbytes" : 4959988,
"bw_bytes" : 725471748,
"bw" : 708468,
"iops" : 177116.554778,
"runtime" : 7001,
"total_ios" : 1239993,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1432,
"max" : 25278,
"mean" : 2296.353298,
"stddev" : 1460.013791,
"N" : 1239994
},
"clat_ns" : {
"min" : 5000,
"max" : 315411,
"mean" : 19955.888521,
"stddev" : 5758.486187,
"N" : 1239996,
"percentile" : {
"1.000000" : 14656,
"5.000000" : 15808,
"10.000000" : 15808,
"20.000000" : 16064,
"30.000000" : 17280,
"40.000000" : 17536,
"50.000000" : 19328,
"60.000000" : 19584,
"70.000000" : 21376,
"80.000000" : 21632,
"90.000000" : 21888,
"95.000000" : 31360,
"99.000000" : 43264,
"99.500000" : 46336,
"99.900000" : 76288,
"99.950000" : 92672,
"99.990000" : 105984
}
},
"lat_ns" : {
"min" : 10500,
"max" : 316904,
"mean" : 22252.243399,
"stddev" : 5586.706939,
"N" : 1239996
},
"bw_min" : 704384,
"bw_max" : 710608,
"bw_agg" : 100.000000,
"bw_mean" : 708637.857143,
"bw_dev" : 1613.810486,
"bw_samples" : 14,
"iops_min" : 176096,
"iops_max" : 177652,
"iops_mean" : 177159.285714,
"iops_stddev" : 403.671311,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 16.328571,
"sys_cpu" : 49.528571,
"ctx" : 320288,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.015161,
"20" : 66.261019,
"50" : 33.495592,
"100" : 0.202824,
"250" : 0.025323,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=88647: Tue Oct 25 23:46:32 2022
read: IOPS=177k, BW=692MiB/s (725MB/s)(4844MiB/7001msec)
slat (nsec): min=1432, max=25278, avg=2296.35, stdev=1460.01
clat (usec): min=5, max=315, avg=19.96, stdev= 5.76
lat (usec): min=10, max=316, avg=22.25, stdev= 5.59
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17],
| 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 20],
| 70.00th=[ 22], 80.00th=[ 22], 90.00th=[ 22], 95.00th=[ 32],
| 99.00th=[ 44], 99.50th=[ 47], 99.90th=[ 77], 99.95th=[ 93],
| 99.99th=[ 106]
bw ( KiB/s): min=704384, max=710608, per=100.00%, avg=708637.86, stdev=1613.81, samples=14
iops : min=176096, max=177652, avg=177159.29, stdev=403.67, samples=14
lat (usec) : 10=0.02%, 20=66.26%, 50=33.50%, 100=0.20%, 250=0.03%
lat (usec) : 500=0.01%
cpu : usr=16.33%, sys=49.53%, ctx=320288, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1239993,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=692MiB/s (725MB/s), 692MiB/s-692MiB/s (725MB/s-725MB/s), io=4844MiB (5079MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_4_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-xnvme_2.txt"
$ cat /tmp/fio-output_4K_4_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734406,
"timestamp_ms" : 1666734406559,
"time" : "Tue Oct 25 23:46:46 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 5081395200,
"io_kbytes" : 4962300,
"bw_bytes" : 725809912,
"bw" : 708798,
"iops" : 177199.114412,
"runtime" : 7001,
"total_ios" : 1240571,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1402,
"max" : 30307,
"mean" : 2212.269113,
"stddev" : 1350.794632,
"N" : 1240572
},
"clat_ns" : {
"min" : 2756,
"max" : 142718,
"mean" : 20076.305615,
"stddev" : 5924.159167,
"N" : 1240574,
"percentile" : {
"1.000000" : 14144,
"5.000000" : 14784,
"10.000000" : 15040,
"20.000000" : 15936,
"30.000000" : 17280,
"40.000000" : 17536,
"50.000000" : 19328,
"60.000000" : 21376,
"70.000000" : 21632,
"80.000000" : 21888,
"90.000000" : 22144,
"95.000000" : 31360,
"99.000000" : 43776,
"99.500000" : 46336,
"99.900000" : 80384,
"99.950000" : 94720,
"99.990000" : 105984
}
},
"lat_ns" : {
"min" : 10880,
"max" : 144201,
"mean" : 22288.576315,
"stddev" : 6010.338472,
"N" : 1240574
},
"bw_min" : 700656,
"bw_max" : 710264,
"bw_agg" : 100.000000,
"bw_mean" : 708977.357143,
"bw_dev" : 2422.535722,
"bw_samples" : 14,
"iops_min" : 175164,
"iops_max" : 177566,
"iops_mean" : 177244.428571,
"iops_stddev" : 605.618910,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.228571,
"sys_cpu" : 50.371429,
"ctx" : 318778,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.026842,
"20" : 57.241706,
"50" : 42.487371,
"100" : 0.212402,
"250" : 0.031760,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=88686: Tue Oct 25 23:46:46 2022
read: IOPS=177k, BW=692MiB/s (726MB/s)(4846MiB/7001msec)
slat (nsec): min=1402, max=30307, avg=2212.27, stdev=1350.79
clat (usec): min=2, max=142, avg=20.08, stdev= 5.92
lat (usec): min=10, max=144, avg=22.29, stdev= 6.01
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 16],
| 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 22],
| 70.00th=[ 22], 80.00th=[ 22], 90.00th=[ 23], 95.00th=[ 32],
| 99.00th=[ 44], 99.50th=[ 47], 99.90th=[ 81], 99.95th=[ 95],
| 99.99th=[ 106]
bw ( KiB/s): min=700656, max=710264, per=100.00%, avg=708977.36, stdev=2422.54, samples=14
iops : min=175164, max=177566, avg=177244.43, stdev=605.62, samples=14
lat (usec) : 4=0.01%, 10=0.03%, 20=57.24%, 50=42.49%, 100=0.21%
lat (usec) : 250=0.03%
cpu : usr=14.23%, sys=50.37%, ctx=318778, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1240571,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=692MiB/s (726MB/s), 692MiB/s-692MiB/s (726MB/s-726MB/s), io=4846MiB (5081MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt"
$ cat /tmp/fio-output_4K_1_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733129,
"timestamp_ms" : 1666733129789,
"time" : "Tue Oct 25 23:25:29 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 1804812288,
"io_kbytes" : 1762512,
"bw_bytes" : 257793499,
"bw" : 251751,
"iops" : 62937.723182,
"runtime" : 7001,
"total_ios" : 440627,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2425,
"max" : 42410,
"mean" : 2632.123117,
"stddev" : 277.348007,
"N" : 440628
},
"clat_ns" : {
"min" : 210,
"max" : 185869,
"mean" : 12588.789237,
"stddev" : 2063.603963,
"N" : 440627,
"percentile" : {
"1.000000" : 12096,
"5.000000" : 12224,
"10.000000" : 12224,
"20.000000" : 12224,
"30.000000" : 12224,
"40.000000" : 12352,
"50.000000" : 12352,
"60.000000" : 12352,
"70.000000" : 12352,
"80.000000" : 12352,
"90.000000" : 12480,
"95.000000" : 12608,
"99.000000" : 22400,
"99.500000" : 28032,
"99.900000" : 39168,
"99.950000" : 50432,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 12183,
"max" : 188574,
"mean" : 15220.912393,
"stddev" : 2084.834946,
"N" : 440627
},
"bw_min" : 250536,
"bw_max" : 252744,
"bw_agg" : 100.000000,
"bw_mean" : 251847.142857,
"bw_dev" : 664.546445,
"bw_samples" : 14,
"iops_min" : 62634,
"iops_max" : 63186,
"iops_mean" : 62961.714286,
"iops_stddev" : 166.188229,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 9.728571,
"sys_cpu" : 38.085714,
"ctx" : 440609,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 98.589737,
"50" : 1.345809,
"100" : 0.053787,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 630140,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 6063,
"write_ticks" : 0,
"in_queue" : 6063,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=84772: Tue Oct 25 23:25:29 2022
read: IOPS=62.9k, BW=246MiB/s (258MB/s)(1721MiB/7001msec)
slat (nsec): min=2425, max=42410, avg=2632.12, stdev=277.35
clat (nsec): min=210, max=185869, avg=12588.79, stdev=2063.60
lat (usec): min=12, max=188, avg=15.22, stdev= 2.08
clat percentiles (nsec):
| 1.00th=[12096], 5.00th=[12224], 10.00th=[12224], 20.00th=[12224],
| 30.00th=[12224], 40.00th=[12352], 50.00th=[12352], 60.00th=[12352],
| 70.00th=[12352], 80.00th=[12352], 90.00th=[12480], 95.00th=[12608],
| 99.00th=[22400], 99.50th=[28032], 99.90th=[39168], 99.95th=[50432],
| 99.99th=[66048]
bw ( KiB/s): min=250536, max=252744, per=100.00%, avg=251847.14, stdev=664.55, samples=14
iops : min=62634, max=63186, avg=62961.71, stdev=166.19, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=98.59%, 50=1.35%, 100=0.05%
lat (usec) : 250=0.01%
cpu : usr=9.73%, sys=38.09%, ctx=440609, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=440627,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=246MiB/s (258MB/s), 246MiB/s-246MiB/s (258MB/s-258MB/s), io=1721MiB (1805MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=630140/0, merge=0/0, ticks=6063/0, in_queue=6063, util=99.09%
$ cat /tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734420,
"timestamp_ms" : 1666734420189,
"time" : "Tue Oct 25 23:47:00 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 6420176896,
"io_kbytes" : 6269704,
"bw_bytes" : 917037122,
"bw" : 895544,
"iops" : 223884.873589,
"runtime" : 7001,
"total_ios" : 1567418,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1082,
"max" : 24687,
"mean" : 1805.728822,
"stddev" : 832.327889,
"N" : 1567419
},
"clat_ns" : {
"min" : 11222,
"max" : 215775,
"mean" : 33639.284363,
"stddev" : 8658.611942,
"N" : 1567425,
"percentile" : {
"1.000000" : 24960,
"5.000000" : 28800,
"10.000000" : 29568,
"20.000000" : 30080,
"30.000000" : 30080,
"40.000000" : 30848,
"50.000000" : 31360,
"60.000000" : 31616,
"70.000000" : 31872,
"80.000000" : 32640,
"90.000000" : 45312,
"95.000000" : 52480,
"99.000000" : 62720,
"99.500000" : 67072,
"99.900000" : 127488,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 15389,
"max" : 218299,
"mean" : 35445.013616,
"stddev" : 8573.177387,
"N" : 1567425
},
"bw_min" : 893760,
"bw_max" : 898008,
"bw_agg" : 100.000000,
"bw_mean" : 895765.357143,
"bw_dev" : 1348.294627,
"bw_samples" : 14,
"iops_min" : 223440,
"iops_max" : 224500,
"iops_mean" : 223941.285714,
"iops_stddev" : 336.789920,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.514286,
"sys_cpu" : 58.171429,
"ctx" : 756092,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999936,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.102462,
"50" : 91.998178,
"100" : 7.644419,
"250" : 0.255388,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=88726: Tue Oct 25 23:47:00 2022
read: IOPS=224k, BW=875MiB/s (917MB/s)(6123MiB/7001msec)
slat (nsec): min=1082, max=24687, avg=1805.73, stdev=832.33
clat (usec): min=11, max=215, avg=33.64, stdev= 8.66
lat (usec): min=15, max=218, avg=35.45, stdev= 8.57
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 30], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 46], 95.00th=[ 53],
| 99.00th=[ 63], 99.50th=[ 68], 99.90th=[ 128], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=893760, max=898008, per=100.00%, avg=895765.36, stdev=1348.29, samples=14
iops : min=223440, max=224500, avg=223941.29, stdev=336.79, samples=14
lat (usec) : 20=0.10%, 50=92.00%, 100=7.64%, 250=0.26%
cpu : usr=14.51%, sys=58.17%, ctx=756092, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567418,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6123MiB (6420MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt"
$ cat /tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734433,
"timestamp_ms" : 1666734433845,
"time" : "Tue Oct 25 23:47:13 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 6416601088,
"io_kbytes" : 6266212,
"bw_bytes" : 916526365,
"bw" : 895045,
"iops" : 223760.319954,
"runtime" : 7001,
"total_ios" : 1566546,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1102,
"max" : 42740,
"mean" : 1833.665654,
"stddev" : 837.174942,
"N" : 1566546
},
"clat_ns" : {
"min" : 4929,
"max" : 212439,
"mean" : 33633.179418,
"stddev" : 8764.799429,
"N" : 1566553,
"percentile" : {
"1.000000" : 24704,
"5.000000" : 28800,
"10.000000" : 29568,
"20.000000" : 30080,
"30.000000" : 30080,
"40.000000" : 30592,
"50.000000" : 31360,
"60.000000" : 31616,
"70.000000" : 31872,
"80.000000" : 32640,
"90.000000" : 45312,
"95.000000" : 52480,
"99.000000" : 62720,
"99.500000" : 68096,
"99.900000" : 130560,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 15479,
"max" : 215695,
"mean" : 35466.844208,
"stddev" : 8680.017687,
"N" : 1566553
},
"bw_min" : 891968,
"bw_max" : 897104,
"bw_agg" : 100.000000,
"bw_mean" : 895253.928571,
"bw_dev" : 1517.357950,
"bw_samples" : 14,
"iops_min" : 222992,
"iops_max" : 224276,
"iops_mean" : 223813.428571,
"iops_stddev" : 379.390141,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.785714,
"sys_cpu" : 58.928571,
"ctx" : 753694,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999936,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 0.106285,
"50" : 92.040195,
"100" : 7.578456,
"250" : 0.275447,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=88764: Tue Oct 25 23:47:13 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6119MiB/7001msec)
slat (nsec): min=1102, max=42740, avg=1833.67, stdev=837.17
clat (usec): min=4, max=212, avg=33.63, stdev= 8.76
lat (usec): min=15, max=215, avg=35.47, stdev= 8.68
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 30], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 46], 95.00th=[ 53],
| 99.00th=[ 63], 99.50th=[ 69], 99.90th=[ 131], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=891968, max=897104, per=100.00%, avg=895253.93, stdev=1517.36, samples=14
iops : min=222992, max=224276, avg=223813.43, stdev=379.39, samples=14
lat (usec) : 10=0.01%, 20=0.11%, 50=92.04%, 100=7.58%, 250=0.28%
cpu : usr=13.79%, sys=58.93%, ctx=753694, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1566546,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6119MiB (6417MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt"
$ cat /tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734447,
"timestamp_ms" : 1666734447509,
"time" : "Tue Oct 25 23:47:27 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 6417027072,
"io_kbytes" : 6266628,
"bw_bytes" : 916587212,
"bw" : 895104,
"iops" : 223775.032138,
"runtime" : 7001,
"total_ios" : 1566649,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1082,
"max" : 28684,
"mean" : 1801.328699,
"stddev" : 837.999667,
"N" : 1566650
},
"clat_ns" : {
"min" : 11131,
"max" : 206056,
"mean" : 33649.558555,
"stddev" : 8666.871644,
"N" : 1566656,
"percentile" : {
"1.000000" : 24704,
"5.000000" : 28800,
"10.000000" : 29824,
"20.000000" : 30080,
"30.000000" : 30336,
"40.000000" : 30848,
"50.000000" : 31616,
"60.000000" : 31616,
"70.000000" : 31872,
"80.000000" : 32640,
"90.000000" : 45312,
"95.000000" : 51968,
"99.000000" : 62720,
"99.500000" : 67072,
"99.900000" : 128512,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 15359,
"max" : 208260,
"mean" : 35450.887665,
"stddev" : 8587.323029,
"N" : 1566656
},
"bw_min" : 885416,
"bw_max" : 897224,
"bw_agg" : 100.000000,
"bw_mean" : 895327.500000,
"bw_dev" : 3021.324589,
"bw_samples" : 14,
"iops_min" : 221354,
"iops_max" : 224306,
"iops_mean" : 223831.857143,
"iops_stddev" : 755.321415,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.185714,
"sys_cpu" : 58.757143,
"ctx" : 755492,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999936,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.092554,
"50" : 92.064145,
"100" : 7.584341,
"250" : 0.259407,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=88804: Tue Oct 25 23:47:27 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6120MiB/7001msec)
slat (nsec): min=1082, max=28684, avg=1801.33, stdev=838.00
clat (usec): min=11, max=206, avg=33.65, stdev= 8.67
lat (usec): min=15, max=208, avg=35.45, stdev= 8.59
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 30], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 46], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 68], 99.90th=[ 129], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=885416, max=897224, per=100.00%, avg=895327.50, stdev=3021.32, samples=14
iops : min=221354, max=224306, avg=223831.86, stdev=755.32, samples=14
lat (usec) : 20=0.09%, 50=92.06%, 100=7.58%, 250=0.26%
cpu : usr=14.19%, sys=58.76%, ctx=755492, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1566649,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6120MiB (6417MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt"
$ cat /tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734461,
"timestamp_ms" : 1666734461202,
"time" : "Tue Oct 25 23:47:41 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 6421319680,
"io_kbytes" : 6270820,
"bw_bytes" : 917200354,
"bw" : 895703,
"iops" : 223924.725039,
"runtime" : 7001,
"total_ios" : 1567697,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1152,
"max" : 175619,
"mean" : 1747.089774,
"stddev" : 791.058747,
"N" : 1567698
},
"clat_ns" : {
"min" : 2465,
"max" : 215894,
"mean" : 33673.439684,
"stddev" : 8679.324358,
"N" : 1567704,
"percentile" : {
"1.000000" : 24960,
"5.000000" : 28800,
"10.000000" : 29056,
"20.000000" : 29824,
"30.000000" : 30336,
"40.000000" : 31104,
"50.000000" : 31616,
"60.000000" : 31872,
"70.000000" : 32384,
"80.000000" : 33024,
"90.000000" : 45312,
"95.000000" : 51968,
"99.000000" : 62720,
"99.500000" : 66048,
"99.900000" : 129536,
"99.950000" : 140288,
"99.990000" : 158720
}
},
"lat_ns" : {
"min" : 15198,
"max" : 217137,
"mean" : 35420.530389,
"stddev" : 8614.100339,
"N" : 1567704
},
"bw_min" : 894104,
"bw_max" : 897792,
"bw_agg" : 100.000000,
"bw_mean" : 895917.571429,
"bw_dev" : 1033.548683,
"bw_samples" : 14,
"iops_min" : 223526,
"iops_max" : 224448,
"iops_mean" : 223979.500000,
"iops_stddev" : 258.444087,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.142857,
"sys_cpu" : 56.985714,
"ctx" : 712649,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 0.070932,
"50" : 92.202383,
"100" : 7.466940,
"250" : 0.259999,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=88843: Tue Oct 25 23:47:41 2022
read: IOPS=224k, BW=875MiB/s (917MB/s)(6124MiB/7001msec)
slat (nsec): min=1152, max=175619, avg=1747.09, stdev=791.06
clat (usec): min=2, max=215, avg=33.67, stdev= 8.68
lat (usec): min=15, max=217, avg=35.42, stdev= 8.61
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 30], 20.00th=[ 30],
| 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 33], 80.00th=[ 34], 90.00th=[ 46], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 67], 99.90th=[ 130], 99.95th=[ 141],
| 99.99th=[ 159]
bw ( KiB/s): min=894104, max=897792, per=100.00%, avg=895917.57, stdev=1033.55, samples=14
iops : min=223526, max=224448, avg=223979.50, stdev=258.44, samples=14
lat (usec) : 4=0.01%, 10=0.01%, 20=0.07%, 50=92.20%, 100=7.47%
lat (usec) : 250=0.26%
cpu : usr=17.14%, sys=56.99%, ctx=712649, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567697,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6124MiB (6421MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-reference_1.txt': No such file or directory
$ rm /tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt"
$ cat /tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734474,
"timestamp_ms" : 1666734474814,
"time" : "Tue Oct 25 23:47:54 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 6416293888,
"io_kbytes" : 6265912,
"bw_bytes" : 916482486,
"bw" : 895002,
"iops" : 223749.607199,
"runtime" : 7001,
"total_ios" : 1566471,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1192,
"max" : 40917,
"mean" : 1637.510650,
"stddev" : 539.612945,
"N" : 1566471
},
"clat_ns" : {
"min" : 221,
"max" : 212459,
"mean" : 33807.519823,
"stddev" : 8996.658594,
"N" : 1566478,
"percentile" : {
"1.000000" : 23936,
"5.000000" : 27520,
"10.000000" : 28800,
"20.000000" : 29568,
"30.000000" : 30592,
"40.000000" : 31104,
"50.000000" : 31616,
"60.000000" : 32128,
"70.000000" : 32640,
"80.000000" : 33536,
"90.000000" : 45312,
"95.000000" : 52480,
"99.000000" : 62720,
"99.500000" : 68096,
"99.900000" : 134144,
"99.950000" : 144384,
"99.990000" : 160768
}
},
"lat_ns" : {
"min" : 14056,
"max" : 213731,
"mean" : 35445.030097,
"stddev" : 8951.929801,
"N" : 1566478
},
"bw_min" : 888056,
"bw_max" : 897410,
"bw_agg" : 100.000000,
"bw_mean" : 895350.500000,
"bw_dev" : 2293.493498,
"bw_samples" : 14,
"iops_min" : 222014,
"iops_max" : 224352,
"iops_mean" : 223837.571429,
"iops_stddev" : 573.327228,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.571429,
"sys_cpu" : 55.157143,
"ctx" : 696395,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 0.234029,
"50" : 91.735755,
"100" : 7.758267,
"250" : 0.272204,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=88881: Tue Oct 25 23:47:54 2022
read: IOPS=224k, BW=874MiB/s (916MB/s)(6119MiB/7001msec)
slat (nsec): min=1192, max=40917, avg=1637.51, stdev=539.61
clat (nsec): min=221, max=212459, avg=33807.52, stdev=8996.66
lat (usec): min=14, max=213, avg=35.45, stdev= 8.95
clat percentiles (usec):
| 1.00th=[ 24], 5.00th=[ 28], 10.00th=[ 29], 20.00th=[ 30],
| 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 33],
| 70.00th=[ 33], 80.00th=[ 34], 90.00th=[ 46], 95.00th=[ 53],
| 99.00th=[ 63], 99.50th=[ 69], 99.90th=[ 135], 99.95th=[ 145],
| 99.99th=[ 161]
bw ( KiB/s): min=888056, max=897410, per=100.00%, avg=895350.50, stdev=2293.49, samples=14
iops : min=222014, max=224352, avg=223837.57, stdev=573.33, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=0.23%, 50=91.74%, 100=7.76%
lat (usec) : 250=0.27%
cpu : usr=17.57%, sys=55.16%, ctx=696395, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1566471,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (916MB/s), 874MiB/s-874MiB/s (916MB/s-916MB/s), io=6119MiB (6416MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt"
$ cat /tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734488,
"timestamp_ms" : 1666734488432,
"time" : "Tue Oct 25 23:48:08 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1"
},
"read" : {
"io_bytes" : 6412873728,
"io_kbytes" : 6262572,
"bw_bytes" : 915993962,
"bw" : 894525,
"iops" : 223630.195686,
"runtime" : 7001,
"total_ios" : 1565635,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1132,
"max" : 29926,
"mean" : 1678.465054,
"stddev" : 744.939201,
"N" : 1565636
},
"clat_ns" : {
"min" : 7093,
"max" : 395762,
"mean" : 33756.134564,
"stddev" : 8803.715628,
"N" : 1565642,
"percentile" : {
"1.000000" : 24960,
"5.000000" : 28800,
"10.000000" : 29056,
"20.000000" : 30080,
"30.000000" : 30592,
"40.000000" : 31104,
"50.000000" : 31616,
"60.000000" : 31616,
"70.000000" : 32128,
"80.000000" : 33024,
"90.000000" : 45824,
"95.000000" : 51968,
"99.000000" : 62720,
"99.500000" : 68096,
"99.900000" : 132096,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 15138,
"max" : 398527,
"mean" : 35434.600653,
"stddev" : 8758.561285,
"N" : 1565642
},
"bw_min" : 888520,
"bw_max" : 898120,
"bw_agg" : 100.000000,
"bw_mean" : 894740.571429,
"bw_dev" : 2660.843987,
"bw_samples" : 14,
"iops_min" : 222130,
"iops_max" : 224530,
"iops_mean" : 223685.142857,
"iops_stddev" : 665.288700,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.742857,
"sys_cpu" : 55.314286,
"ctx" : 725537,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 0.082842,
"50" : 92.137439,
"100" : 7.499130,
"250" : 0.280461,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=88919: Tue Oct 25 23:48:08 2022
read: IOPS=224k, BW=874MiB/s (916MB/s)(6116MiB/7001msec)
slat (nsec): min=1132, max=29926, avg=1678.47, stdev=744.94
clat (usec): min=7, max=395, avg=33.76, stdev= 8.80
lat (usec): min=15, max=398, avg=35.43, stdev= 8.76
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 30], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 33], 80.00th=[ 34], 90.00th=[ 46], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 69], 99.90th=[ 133], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=888520, max=898120, per=100.00%, avg=894740.57, stdev=2660.84, samples=14
iops : min=222130, max=224530, avg=223685.14, stdev=665.29, samples=14
lat (usec) : 10=0.01%, 20=0.08%, 50=92.14%, 100=7.50%, 250=0.28%
lat (usec) : 500=0.01%
cpu : usr=17.74%, sys=55.31%, ctx=725537, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1565635,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (916MB/s), 874MiB/s-874MiB/s (916MB/s-916MB/s), io=6116MiB (6413MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734503,
"timestamp_ms" : 1666734503475,
"time" : "Tue Oct 25 23:48:23 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6421094400,
"io_kbytes" : 6270600,
"bw_bytes" : 917168175,
"bw" : 895672,
"iops" : 223917.011855,
"runtime" : 7001,
"total_ios" : 1567643,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 661,
"max" : 38231,
"mean" : 991.606545,
"stddev" : 179.577583,
"N" : 1567643
},
"clat_ns" : {
"min" : 1854,
"max" : 230002,
"mean" : 34445.715295,
"stddev" : 8475.141091,
"N" : 1567649,
"percentile" : {
"1.000000" : 28032,
"5.000000" : 30080,
"10.000000" : 30592,
"20.000000" : 31104,
"30.000000" : 31360,
"40.000000" : 31616,
"50.000000" : 31872,
"60.000000" : 32384,
"70.000000" : 32640,
"80.000000" : 33536,
"90.000000" : 44288,
"95.000000" : 52480,
"99.000000" : 62720,
"99.500000" : 66048,
"99.900000" : 129536,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 13204,
"max" : 230923,
"mean" : 35437.321713,
"stddev" : 8475.876232,
"N" : 1567649
},
"bw_min" : 893891,
"bw_max" : 897088,
"bw_agg" : 100.000000,
"bw_mean" : 895900.214286,
"bw_dev" : 990.139709,
"bw_samples" : 14,
"iops_min" : 223472,
"iops_max" : 224272,
"iops_mean" : 223975.000000,
"iops_stddev" : 247.652051,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 18,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.100000,
"4" : 0.720955,
"8" : 99.261694,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.995006,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 0.022008,
"50" : 91.171332,
"100" : 8.551245,
"250" : 0.255160,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=88970: Tue Oct 25 23:48:23 2022
read: IOPS=224k, BW=875MiB/s (917MB/s)(6124MiB/7001msec)
slat (nsec): min=661, max=38231, avg=991.61, stdev=179.58
clat (nsec): min=1854, max=230002, avg=34445.72, stdev=8475.14
lat (usec): min=13, max=230, avg=35.44, stdev= 8.48
clat percentiles (usec):
| 1.00th=[ 29], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32],
| 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 33],
| 70.00th=[ 33], 80.00th=[ 34], 90.00th=[ 45], 95.00th=[ 53],
| 99.00th=[ 63], 99.50th=[ 67], 99.90th=[ 130], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=893891, max=897088, per=100.00%, avg=895900.21, stdev=990.14, samples=14
iops : min=223472, max=224272, avg=223975.00, stdev=247.65, samples=14
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=91.17%
lat (usec) : 100=8.55%, 250=0.26%
cpu : usr=99.99%, sys=0.00%, ctx=18, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.7%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567643,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6124MiB (6421MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-reference_1.txt"
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734518,
"timestamp_ms" : 1666734518563,
"time" : "Tue Oct 25 23:48:38 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6416740352,
"io_kbytes" : 6266348,
"bw_bytes" : 916546257,
"bw" : 895064,
"iops" : 223765.033567,
"runtime" : 7001,
"total_ios" : 1566579,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 651,
"max" : 10229,
"mean" : 922.635981,
"stddev" : 74.486330,
"N" : 1566580
},
"clat_ns" : {
"min" : 13977,
"max" : 229741,
"mean" : 34515.006953,
"stddev" : 8893.514425,
"N" : 1566582,
"percentile" : {
"1.000000" : 24192,
"5.000000" : 30080,
"10.000000" : 30336,
"20.000000" : 30592,
"30.000000" : 31104,
"40.000000" : 31872,
"50.000000" : 32384,
"60.000000" : 32640,
"70.000000" : 33024,
"80.000000" : 33536,
"90.000000" : 45312,
"95.000000" : 52480,
"99.000000" : 63744,
"99.500000" : 69120,
"99.900000" : 136192,
"99.950000" : 144384,
"99.990000" : 160768
}
},
"lat_ns" : {
"min" : 14748,
"max" : 230652,
"mean" : 35437.642907,
"stddev" : 8893.112545,
"N" : 1566582
},
"bw_min" : 892728,
"bw_max" : 898024,
"bw_agg" : 100.000000,
"bw_mean" : 895313.500000,
"bw_dev" : 1341.825436,
"bw_samples" : 14,
"iops_min" : 223182,
"iops_max" : 224506,
"iops_mean" : 223828.357143,
"iops_stddev" : 335.467895,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 4,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.100000,
"4" : 0.602140,
"8" : 99.388540,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.995614,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.048769,
"50" : 91.253298,
"100" : 8.411577,
"250" : 0.286548,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=89020: Tue Oct 25 23:48:38 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6119MiB/7001msec)
slat (nsec): min=651, max=10229, avg=922.64, stdev=74.49
clat (usec): min=13, max=229, avg=34.52, stdev= 8.89
lat (usec): min=14, max=230, avg=35.44, stdev= 8.89
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31],
| 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 33], 60.00th=[ 33],
| 70.00th=[ 34], 80.00th=[ 34], 90.00th=[ 46], 95.00th=[ 53],
| 99.00th=[ 64], 99.50th=[ 70], 99.90th=[ 137], 99.95th=[ 145],
| 99.99th=[ 161]
bw ( KiB/s): min=892728, max=898024, per=100.00%, avg=895313.50, stdev=1341.83, samples=14
iops : min=223182, max=224506, avg=223828.36, stdev=335.47, samples=14
lat (usec) : 20=0.05%, 50=91.25%, 100=8.41%, 250=0.29%
cpu : usr=99.99%, sys=0.00%, ctx=4, majf=0, minf=0
IO depths : 1=0.0%, 2=0.1%, 4=0.6%, 8=99.4%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1566579,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6119MiB (6417MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734533,
"timestamp_ms" : 1666734533650,
"time" : "Tue Oct 25 23:48:53 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6419181568,
"io_kbytes" : 6268732,
"bw_bytes" : 916894953,
"bw" : 895405,
"iops" : 223850.307099,
"runtime" : 7001,
"total_ios" : 1567176,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 651,
"max" : 7023,
"mean" : 994.223934,
"stddev" : 136.149078,
"N" : 1567176
},
"clat_ns" : {
"min" : 11421,
"max" : 178565,
"mean" : 34501.146111,
"stddev" : 8517.845352,
"N" : 1567183,
"percentile" : {
"1.000000" : 28544,
"5.000000" : 30080,
"10.000000" : 30592,
"20.000000" : 31104,
"30.000000" : 31616,
"40.000000" : 31872,
"50.000000" : 31872,
"60.000000" : 32384,
"70.000000" : 32640,
"80.000000" : 33536,
"90.000000" : 44288,
"95.000000" : 52480,
"99.000000" : 63232,
"99.500000" : 66048,
"99.900000" : 130560,
"99.950000" : 140288,
"99.990000" : 154624
}
},
"lat_ns" : {
"min" : 12323,
"max" : 179727,
"mean" : 35495.369933,
"stddev" : 8518.963941,
"N" : 1567183
},
"bw_min" : 892608,
"bw_max" : 897816,
"bw_agg" : 100.000000,
"bw_mean" : 895630.214286,
"bw_dev" : 1342.108684,
"bw_samples" : 14,
"iops_min" : 223152,
"iops_max" : 224454,
"iops_mean" : 223907.500000,
"iops_stddev" : 335.521351,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 4,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.100000,
"4" : 0.630050,
"8" : 99.368801,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.997888,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.013847,
"50" : 91.109358,
"100" : 8.605415,
"250" : 0.271827,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=89070: Tue Oct 25 23:48:53 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6122MiB/7001msec)
slat (nsec): min=651, max=7023, avg=994.22, stdev=136.15
clat (usec): min=11, max=178, avg=34.50, stdev= 8.52
lat (usec): min=12, max=179, avg=35.50, stdev= 8.52
clat percentiles (usec):
| 1.00th=[ 29], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32],
| 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 33],
| 70.00th=[ 33], 80.00th=[ 34], 90.00th=[ 45], 95.00th=[ 53],
| 99.00th=[ 64], 99.50th=[ 67], 99.90th=[ 131], 99.95th=[ 141],
| 99.99th=[ 155]
bw ( KiB/s): min=892608, max=897816, per=100.00%, avg=895630.21, stdev=1342.11, samples=14
iops : min=223152, max=224454, avg=223907.50, stdev=335.52, samples=14
lat (usec) : 20=0.01%, 50=91.11%, 100=8.61%, 250=0.27%
cpu : usr=99.99%, sys=0.00%, ctx=4, majf=0, minf=0
IO depths : 1=0.0%, 2=0.1%, 4=0.6%, 8=99.4%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567176,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6122MiB (6419MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-reference_0.txt"
$ cat /tmp/fio-output_4K_8_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734547,
"timestamp_ms" : 1666734547398,
"time" : "Tue Oct 25 23:49:07 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 6418763776,
"io_kbytes" : 6268324,
"bw_bytes" : 916835277,
"bw" : 895346,
"iops" : 223835.594915,
"runtime" : 7001,
"total_ios" : 1567073,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1222,
"max" : 43772,
"mean" : 1732.981963,
"stddev" : 738.776513,
"N" : 1567074
},
"clat_ns" : {
"min" : 80,
"max" : 210004,
"mean" : 33654.755580,
"stddev" : 8683.484113,
"N" : 1567080,
"percentile" : {
"1.000000" : 24704,
"5.000000" : 28544,
"10.000000" : 28800,
"20.000000" : 29568,
"30.000000" : 30592,
"40.000000" : 31104,
"50.000000" : 31360,
"60.000000" : 31872,
"70.000000" : 32384,
"80.000000" : 33024,
"90.000000" : 45312,
"95.000000" : 51968,
"99.000000" : 62720,
"99.500000" : 66048,
"99.900000" : 128512,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 15339,
"max" : 211277,
"mean" : 35387.737334,
"stddev" : 8632.811278,
"N" : 1567080
},
"bw_min" : 892488,
"bw_max" : 898184,
"bw_agg" : 100.000000,
"bw_mean" : 895707.214286,
"bw_dev" : 1505.889371,
"bw_samples" : 14,
"iops_min" : 223122,
"iops_max" : 224546,
"iops_mean" : 223926.928571,
"iops_stddev" : 376.422525,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 16.885714,
"sys_cpu" : 58.414286,
"ctx" : 669792,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999936,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 0.103697,
"50" : 92.161693,
"100" : 7.477826,
"250" : 0.257040,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2235843,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 70932,
"write_ticks" : 0,
"in_queue" : 70932,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=89109: Tue Oct 25 23:49:07 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6121MiB/7001msec)
slat (nsec): min=1222, max=43772, avg=1732.98, stdev=738.78
clat (nsec): min=80, max=210004, avg=33654.76, stdev=8683.48
lat (usec): min=15, max=211, avg=35.39, stdev= 8.63
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 30],
| 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 33], 80.00th=[ 34], 90.00th=[ 46], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 67], 99.90th=[ 129], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=892488, max=898184, per=100.00%, avg=895707.21, stdev=1505.89, samples=14
iops : min=223122, max=224546, avg=223926.93, stdev=376.42, samples=14
lat (nsec) : 100=0.01%
lat (usec) : 10=0.01%, 20=0.10%, 50=92.16%, 100=7.48%, 250=0.26%
cpu : usr=16.89%, sys=58.41%, ctx=669792, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567073,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6121MiB (6419MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2235843/0, merge=0/0, ticks=70932/0, in_queue=70932, util=99.09%
$ rm /tmp/fio-output_4K_8_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-reference_1.txt"
$ cat /tmp/fio-output_4K_1_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733143,
"timestamp_ms" : 1666733143402,
"time" : "Tue Oct 25 23:25:43 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 1824374784,
"io_kbytes" : 1781616,
"bw_bytes" : 260587742,
"bw" : 254480,
"iops" : 63620.054278,
"runtime" : 7001,
"total_ios" : 445404,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2424,
"max" : 34645,
"mean" : 2592.826618,
"stddev" : 222.766188,
"N" : 445404
},
"clat_ns" : {
"min" : 240,
"max" : 122219,
"mean" : 12494.563899,
"stddev" : 2015.777434,
"N" : 445404,
"percentile" : {
"1.000000" : 11968,
"5.000000" : 11968,
"10.000000" : 11968,
"20.000000" : 12096,
"30.000000" : 12096,
"40.000000" : 12224,
"50.000000" : 12224,
"60.000000" : 12352,
"70.000000" : 12352,
"80.000000" : 12352,
"90.000000" : 12480,
"95.000000" : 12480,
"99.000000" : 22400,
"99.500000" : 27776,
"99.900000" : 33024,
"99.950000" : 49920,
"99.990000" : 64768
}
},
"lat_ns" : {
"min" : 12434,
"max" : 124784,
"mean" : 15087.390517,
"stddev" : 2032.769095,
"N" : 445404
},
"bw_min" : 253272,
"bw_max" : 255968,
"bw_agg" : 100.000000,
"bw_mean" : 254578.357143,
"bw_dev" : 701.625103,
"bw_samples" : 14,
"iops_min" : 63318,
"iops_max" : 63992,
"iops_mean" : 63644.571429,
"iops_stddev" : 175.405778,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 11.200000,
"sys_cpu" : 36.728571,
"ctx" : 445388,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 98.600821,
"50" : 1.339009,
"100" : 0.052537,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 638282,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 6018,
"write_ticks" : 0,
"in_queue" : 6018,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=84810: Tue Oct 25 23:25:43 2022
read: IOPS=63.6k, BW=249MiB/s (261MB/s)(1740MiB/7001msec)
slat (nsec): min=2424, max=34645, avg=2592.83, stdev=222.77
clat (nsec): min=240, max=122219, avg=12494.56, stdev=2015.78
lat (usec): min=12, max=124, avg=15.09, stdev= 2.03
clat percentiles (nsec):
| 1.00th=[11968], 5.00th=[11968], 10.00th=[11968], 20.00th=[12096],
| 30.00th=[12096], 40.00th=[12224], 50.00th=[12224], 60.00th=[12352],
| 70.00th=[12352], 80.00th=[12352], 90.00th=[12480], 95.00th=[12480],
| 99.00th=[22400], 99.50th=[27776], 99.90th=[33024], 99.95th=[49920],
| 99.99th=[64768]
bw ( KiB/s): min=253272, max=255968, per=100.00%, avg=254578.36, stdev=701.63, samples=14
iops : min=63318, max=63992, avg=63644.57, stdev=175.41, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 10=0.01%, 20=98.60%, 50=1.34%, 100=0.05%, 250=0.01%
cpu : usr=11.20%, sys=36.73%, ctx=445388, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=445404,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=249MiB/s (261MB/s), 249MiB/s-249MiB/s (261MB/s-261MB/s), io=1740MiB (1824MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=638282/0, merge=0/0, ticks=6018/0, in_queue=6018, util=99.09%
$ cat /tmp/fio-output_4K_8_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734561,
"timestamp_ms" : 1666734561035,
"time" : "Tue Oct 25 23:49:21 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 6418571264,
"io_kbytes" : 6268136,
"bw_bytes" : 916807779,
"bw" : 895320,
"iops" : 223828.881588,
"runtime" : 7001,
"total_ios" : 1567026,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1212,
"max" : 36078,
"mean" : 1714.383513,
"stddev" : 722.179076,
"N" : 1567027
},
"clat_ns" : {
"min" : 12804,
"max" : 206107,
"mean" : 33675.974473,
"stddev" : 8691.726209,
"N" : 1567033,
"percentile" : {
"1.000000" : 24960,
"5.000000" : 28544,
"10.000000" : 28800,
"20.000000" : 29824,
"30.000000" : 30592,
"40.000000" : 31104,
"50.000000" : 31360,
"60.000000" : 31616,
"70.000000" : 32384,
"80.000000" : 33024,
"90.000000" : 45312,
"95.000000" : 51968,
"99.000000" : 62720,
"99.500000" : 67072,
"99.900000" : 128512,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 15399,
"max" : 207379,
"mean" : 35390.359061,
"stddev" : 8641.981671,
"N" : 1567033
},
"bw_min" : 893378,
"bw_max" : 897136,
"bw_agg" : 100.000000,
"bw_mean" : 895682.857143,
"bw_dev" : 1185.473801,
"bw_samples" : 14,
"iops_min" : 223344,
"iops_max" : 224284,
"iops_mean" : 223920.642857,
"iops_stddev" : 296.455888,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 16.600000,
"sys_cpu" : 58.057143,
"ctx" : 693034,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999936,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.086406,
"50" : 92.191195,
"100" : 7.458842,
"250" : 0.264003,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2235997,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 71144,
"write_ticks" : 0,
"in_queue" : 71144,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=89147: Tue Oct 25 23:49:21 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6121MiB/7001msec)
slat (nsec): min=1212, max=36078, avg=1714.38, stdev=722.18
clat (usec): min=12, max=206, avg=33.68, stdev= 8.69
lat (usec): min=15, max=207, avg=35.39, stdev= 8.64
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 30],
| 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 33], 80.00th=[ 34], 90.00th=[ 46], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 68], 99.90th=[ 129], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=893378, max=897136, per=100.00%, avg=895682.86, stdev=1185.47, samples=14
iops : min=223344, max=224284, avg=223920.64, stdev=296.46, samples=14
lat (usec) : 20=0.09%, 50=92.19%, 100=7.46%, 250=0.26%
cpu : usr=16.60%, sys=58.06%, ctx=693034, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567026,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6121MiB (6419MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2235997/0, merge=0/0, ticks=71144/0, in_queue=71144, util=99.09%
$ rm /tmp/fio-output_4K_8_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-reference_2.txt"
$ cat /tmp/fio-output_4K_8_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734574,
"timestamp_ms" : 1666734574665,
"time" : "Tue Oct 25 23:49:34 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 6419468288,
"io_kbytes" : 6269012,
"bw_bytes" : 916935907,
"bw" : 895445,
"iops" : 223860.162834,
"runtime" : 7001,
"total_ios" : 1567245,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1232,
"max" : 53460,
"mean" : 1707.207037,
"stddev" : 717.038812,
"N" : 1567246
},
"clat_ns" : {
"min" : 7434,
"max" : 196628,
"mean" : 33672.910071,
"stddev" : 8730.705101,
"N" : 1567252,
"percentile" : {
"1.000000" : 25216,
"5.000000" : 28544,
"10.000000" : 28800,
"20.000000" : 29312,
"30.000000" : 30592,
"40.000000" : 31104,
"50.000000" : 31360,
"60.000000" : 31872,
"70.000000" : 32384,
"80.000000" : 33024,
"90.000000" : 45312,
"95.000000" : 51968,
"99.000000" : 62720,
"99.500000" : 67072,
"99.900000" : 129536,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 15579,
"max" : 198051,
"mean" : 35380.117355,
"stddev" : 8681.842147,
"N" : 1567252
},
"bw_min" : 892529,
"bw_max" : 897192,
"bw_agg" : 100.000000,
"bw_mean" : 895805.071429,
"bw_dev" : 1222.038050,
"bw_samples" : 14,
"iops_min" : 223132,
"iops_max" : 224298,
"iops_mean" : 223951.357143,
"iops_stddev" : 305.643233,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.300000,
"sys_cpu" : 57.928571,
"ctx" : 667803,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999936,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 0.066550,
"50" : 92.209706,
"100" : 7.450845,
"250" : 0.273154,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2236081,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 71107,
"write_ticks" : 0,
"in_queue" : 71107,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=89185: Tue Oct 25 23:49:34 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6122MiB/7001msec)
slat (nsec): min=1232, max=53460, avg=1707.21, stdev=717.04
clat (usec): min=7, max=196, avg=33.67, stdev= 8.73
lat (usec): min=15, max=198, avg=35.38, stdev= 8.68
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 30],
| 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 33], 80.00th=[ 34], 90.00th=[ 46], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 68], 99.90th=[ 130], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=892529, max=897192, per=100.00%, avg=895805.07, stdev=1222.04, samples=14
iops : min=223132, max=224298, avg=223951.36, stdev=305.64, samples=14
lat (usec) : 10=0.01%, 20=0.07%, 50=92.21%, 100=7.45%, 250=0.27%
cpu : usr=17.30%, sys=57.93%, ctx=667803, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567245,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6122MiB (6419MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2236081/0, merge=0/0, ticks=71107/0, in_queue=71107, util=99.09%
$ rm /tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734589,
"timestamp_ms" : 1666734589681,
"time" : "Tue Oct 25 23:49:49 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6419386368,
"io_kbytes" : 6268932,
"bw_bytes" : 916924206,
"bw" : 895433,
"iops" : 223857.448936,
"runtime" : 7001,
"total_ios" : 1567226,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 17783,
"mean" : 177.147500,
"stddev" : 78.228730,
"N" : 1567226
},
"clat_ns" : {
"min" : 13115,
"max" : 1356474,
"mean" : 35300.071299,
"stddev" : 8917.372350,
"N" : 1567233,
"percentile" : {
"1.000000" : 29568,
"5.000000" : 32384,
"10.000000" : 32384,
"20.000000" : 32640,
"30.000000" : 32640,
"40.000000" : 32640,
"50.000000" : 32640,
"60.000000" : 32640,
"70.000000" : 33024,
"80.000000" : 33024,
"90.000000" : 45312,
"95.000000" : 53504,
"99.000000" : 63744,
"99.500000" : 67072,
"99.900000" : 130560,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 13395,
"max" : 1356644,
"mean" : 35477.218807,
"stddev" : 8916.741132,
"N" : 1567233
},
"bw_min" : 893360,
"bw_max" : 897640,
"bw_agg" : 100.000000,
"bw_mean" : 895644.571429,
"bw_dev" : 1100.192830,
"bw_samples" : 14,
"iops_min" : 223340,
"iops_max" : 224410,
"iops_mean" : 223911.142857,
"iops_stddev" : 275.048208,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 39.585714,
"sys_cpu" : 60.385714,
"ctx" : 40,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.100000,
"4" : 1.291773,
"8" : 98.704335,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.996949,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.010000,
"50" : 90.676265,
"100" : 9.055108,
"250" : 0.262374,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=89233: Tue Oct 25 23:49:49 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6122MiB/7001msec)
slat (nsec): min=110, max=17783, avg=177.15, stdev=78.23
clat (usec): min=13, max=1356, avg=35.30, stdev= 8.92
lat (usec): min=13, max=1356, avg=35.48, stdev= 8.92
clat percentiles (usec):
| 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33],
| 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 33],
| 70.00th=[ 34], 80.00th=[ 34], 90.00th=[ 46], 95.00th=[ 54],
| 99.00th=[ 64], 99.50th=[ 68], 99.90th=[ 131], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=893360, max=897640, per=100.00%, avg=895644.57, stdev=1100.19, samples=14
iops : min=223340, max=224410, avg=223911.14, stdev=275.05, samples=14
lat (usec) : 20=0.01%, 50=90.68%, 100=9.06%, 250=0.26%
lat (msec) : 2=0.01%
cpu : usr=39.59%, sys=60.39%, ctx=40, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=1.3%, 8=98.7%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567226,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6122MiB (6419MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734605,
"timestamp_ms" : 1666734605226,
"time" : "Tue Oct 25 23:50:05 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6418595840,
"io_kbytes" : 6268160,
"bw_bytes" : 916811289,
"bw" : 895323,
"iops" : 223829.881446,
"runtime" : 7001,
"total_ios" : 1567033,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 120,
"max" : 18484,
"mean" : 180.659000,
"stddev" : 80.296904,
"N" : 1567033
},
"clat_ns" : {
"min" : 13105,
"max" : 191649,
"mean" : 35285.179822,
"stddev" : 8502.394509,
"N" : 1567040,
"percentile" : {
"1.000000" : 29824,
"5.000000" : 32384,
"10.000000" : 32384,
"20.000000" : 32640,
"30.000000" : 32640,
"40.000000" : 32640,
"50.000000" : 32640,
"60.000000" : 32640,
"70.000000" : 33024,
"80.000000" : 33024,
"90.000000" : 45312,
"95.000000" : 53504,
"99.000000" : 63744,
"99.500000" : 67072,
"99.900000" : 132096,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 13355,
"max" : 191830,
"mean" : 35465.838813,
"stddev" : 8501.391034,
"N" : 1567040
},
"bw_min" : 891984,
"bw_max" : 898280,
"bw_agg" : 100.000000,
"bw_mean" : 895534.142857,
"bw_dev" : 1537.441324,
"bw_samples" : 14,
"iops_min" : 222996,
"iops_max" : 224570,
"iops_mean" : 223883.500000,
"iops_stddev" : 384.365761,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 42.957143,
"sys_cpu" : 57.014286,
"ctx" : 11,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.100000,
"4" : 1.244645,
"8" : 98.750441,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.996629,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.010000,
"50" : 90.670713,
"100" : 9.049012,
"250" : 0.275425,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=89282: Tue Oct 25 23:50:05 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6121MiB/7001msec)
slat (nsec): min=120, max=18484, avg=180.66, stdev=80.30
clat (usec): min=13, max=191, avg=35.29, stdev= 8.50
lat (usec): min=13, max=191, avg=35.47, stdev= 8.50
clat percentiles (usec):
| 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33],
| 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 33],
| 70.00th=[ 34], 80.00th=[ 34], 90.00th=[ 46], 95.00th=[ 54],
| 99.00th=[ 64], 99.50th=[ 68], 99.90th=[ 133], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=891984, max=898280, per=100.00%, avg=895534.14, stdev=1537.44, samples=14
iops : min=222996, max=224570, avg=223883.50, stdev=384.37, samples=14
lat (usec) : 20=0.01%, 50=90.67%, 100=9.05%, 250=0.28%
cpu : usr=42.96%, sys=57.01%, ctx=11, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=1.2%, 8=98.8%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567033,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6121MiB (6419MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-reference_2.txt': No such file or directory
$ rm /tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734620,
"timestamp_ms" : 1666734620429,
"time" : "Tue Oct 25 23:50:20 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6415040512,
"io_kbytes" : 6264688,
"bw_bytes" : 916303458,
"bw" : 894827,
"iops" : 223705.899157,
"runtime" : 7001,
"total_ios" : 1566165,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 41308,
"mean" : 199.295783,
"stddev" : 128.404646,
"N" : 1566165
},
"clat_ns" : {
"min" : 12804,
"max" : 177943,
"mean" : 35310.666389,
"stddev" : 8531.519283,
"N" : 1566172,
"percentile" : {
"1.000000" : 29568,
"5.000000" : 32384,
"10.000000" : 32384,
"20.000000" : 32640,
"30.000000" : 32640,
"40.000000" : 32640,
"50.000000" : 32640,
"60.000000" : 32640,
"70.000000" : 33024,
"80.000000" : 33024,
"90.000000" : 45312,
"95.000000" : 53504,
"99.000000" : 63744,
"99.500000" : 67072,
"99.900000" : 132096,
"99.950000" : 142336,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 13055,
"max" : 178154,
"mean" : 35509.962140,
"stddev" : 8530.220824,
"N" : 1566172
},
"bw_min" : 893232,
"bw_max" : 895976,
"bw_agg" : 100.000000,
"bw_mean" : 895036.428571,
"bw_dev" : 819.653277,
"bw_samples" : 14,
"iops_min" : 223308,
"iops_max" : 223994,
"iops_mean" : 223759.214286,
"iops_stddev" : 205.000630,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 39.642857,
"sys_cpu" : 60.328571,
"ctx" : 19,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.100000,
"4" : 1.353178,
"8" : 98.618983,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.991192,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.011110,
"50" : 90.645749,
"100" : 9.069670,
"250" : 0.273917,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=89331: Tue Oct 25 23:50:20 2022
read: IOPS=224k, BW=874MiB/s (916MB/s)(6118MiB/7001msec)
slat (nsec): min=110, max=41308, avg=199.30, stdev=128.40
clat (usec): min=12, max=177, avg=35.31, stdev= 8.53
lat (usec): min=13, max=178, avg=35.51, stdev= 8.53
clat percentiles (usec):
| 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33],
| 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 33],
| 70.00th=[ 34], 80.00th=[ 34], 90.00th=[ 46], 95.00th=[ 54],
| 99.00th=[ 64], 99.50th=[ 68], 99.90th=[ 133], 99.95th=[ 143],
| 99.99th=[ 157]
bw ( KiB/s): min=893232, max=895976, per=100.00%, avg=895036.43, stdev=819.65, samples=14
iops : min=223308, max=223994, avg=223759.21, stdev=205.00, samples=14
lat (usec) : 20=0.01%, 50=90.65%, 100=9.07%, 250=0.27%
cpu : usr=39.64%, sys=60.33%, ctx=19, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=1.4%, 8=98.6%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1566165,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (916MB/s), 874MiB/s-874MiB/s (916MB/s-916MB/s), io=6118MiB (6415MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734635,
"timestamp_ms" : 1666734635553,
"time" : "Tue Oct 25 23:50:35 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6416384000,
"io_kbytes" : 6266000,
"bw_bytes" : 916495357,
"bw" : 895014,
"iops" : 223752.749607,
"runtime" : 7001,
"total_ios" : 1566493,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1482,
"max" : 67687,
"mean" : 1754.159928,
"stddev" : 246.058703,
"N" : 1566493
},
"clat_ns" : {
"min" : 15098,
"max" : 202079,
"mean" : 33741.697037,
"stddev" : 8649.734313,
"N" : 1566500,
"percentile" : {
"1.000000" : 24448,
"5.000000" : 30848,
"10.000000" : 30848,
"20.000000" : 30848,
"30.000000" : 31104,
"40.000000" : 31104,
"50.000000" : 31104,
"60.000000" : 31104,
"70.000000" : 31360,
"80.000000" : 31360,
"90.000000" : 43776,
"95.000000" : 51968,
"99.000000" : 62208,
"99.500000" : 67072,
"99.900000" : 132096,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 17352,
"max" : 203803,
"mean" : 35495.856783,
"stddev" : 8639.780490,
"N" : 1566500
},
"bw_min" : 894056,
"bw_max" : 897080,
"bw_agg" : 100.000000,
"bw_mean" : 895219.785714,
"bw_dev" : 929.395517,
"bw_samples" : 14,
"iops_min" : 223514,
"iops_max" : 224270,
"iops_mean" : 223804.785714,
"iops_stddev" : 232.359198,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 30.385714,
"sys_cpu" : 69.585714,
"ctx" : 14,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.026684,
"50" : 91.094183,
"100" : 8.585867,
"250" : 0.293713,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=89380: Tue Oct 25 23:50:35 2022
read: IOPS=224k, BW=874MiB/s (916MB/s)(6119MiB/7001msec)
slat (nsec): min=1482, max=67687, avg=1754.16, stdev=246.06
clat (usec): min=15, max=202, avg=33.74, stdev= 8.65
lat (usec): min=17, max=203, avg=35.50, stdev= 8.64
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31],
| 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 44], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 68], 99.90th=[ 133], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=894056, max=897080, per=100.00%, avg=895219.79, stdev=929.40, samples=14
iops : min=223514, max=224270, avg=223804.79, stdev=232.36, samples=14
lat (usec) : 20=0.03%, 50=91.09%, 100=8.59%, 250=0.29%
cpu : usr=30.39%, sys=69.59%, ctx=14, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1566493,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (916MB/s), 874MiB/s-874MiB/s (916MB/s-916MB/s), io=6119MiB (6416MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734650,
"timestamp_ms" : 1666734650706,
"time" : "Tue Oct 25 23:50:50 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6418513920,
"io_kbytes" : 6268080,
"bw_bytes" : 916799588,
"bw" : 895312,
"iops" : 223827.024711,
"runtime" : 7001,
"total_ios" : 1567013,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1473,
"max" : 25368,
"mean" : 1749.800455,
"stddev" : 234.769642,
"N" : 1567013
},
"clat_ns" : {
"min" : 12604,
"max" : 1083192,
"mean" : 33745.384779,
"stddev" : 8890.164459,
"N" : 1567020,
"percentile" : {
"1.000000" : 24448,
"5.000000" : 30848,
"10.000000" : 30848,
"20.000000" : 30848,
"30.000000" : 31104,
"40.000000" : 31104,
"50.000000" : 31104,
"60.000000" : 31104,
"70.000000" : 31360,
"80.000000" : 31360,
"90.000000" : 43776,
"95.000000" : 51968,
"99.000000" : 62208,
"99.500000" : 66048,
"99.900000" : 130560,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 14317,
"max" : 1084925,
"mean" : 35495.185114,
"stddev" : 8880.807151,
"N" : 1567020
},
"bw_min" : 893456,
"bw_max" : 896816,
"bw_agg" : 100.000000,
"bw_mean" : 895527.357143,
"bw_dev" : 964.990044,
"bw_samples" : 14,
"iops_min" : 223364,
"iops_max" : 224204,
"iops_mean" : 223881.785714,
"iops_stddev" : 241.228962,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 25.757143,
"sys_cpu" : 74.214286,
"ctx" : 42,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.046075,
"50" : 91.074356,
"100" : 8.607331,
"250" : 0.272174,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=89429: Tue Oct 25 23:50:50 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6121MiB/7001msec)
slat (nsec): min=1473, max=25368, avg=1749.80, stdev=234.77
clat (usec): min=12, max=1083, avg=33.75, stdev= 8.89
lat (usec): min=14, max=1084, avg=35.50, stdev= 8.88
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31],
| 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 44], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 67], 99.90th=[ 131], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=893456, max=896816, per=100.00%, avg=895527.36, stdev=964.99, samples=14
iops : min=223364, max=224204, avg=223881.79, stdev=241.23, samples=14
lat (usec) : 20=0.05%, 50=91.07%, 100=8.61%, 250=0.27%
lat (msec) : 2=0.01%
cpu : usr=25.76%, sys=74.21%, ctx=42, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567013,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6121MiB (6419MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-reference_2.txt"
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734665,
"timestamp_ms" : 1666734665909,
"time" : "Tue Oct 25 23:51:05 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6418640896,
"io_kbytes" : 6268204,
"bw_bytes" : 916817725,
"bw" : 895329,
"iops" : 223831.452650,
"runtime" : 7001,
"total_ios" : 1567044,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1472,
"max" : 25949,
"mean" : 1754.159593,
"stddev" : 224.643680,
"N" : 1567044
},
"clat_ns" : {
"min" : 15079,
"max" : 220213,
"mean" : 33737.245896,
"stddev" : 8603.637014,
"N" : 1567051,
"percentile" : {
"1.000000" : 24704,
"5.000000" : 30848,
"10.000000" : 30848,
"20.000000" : 30848,
"30.000000" : 31104,
"40.000000" : 31104,
"50.000000" : 31104,
"60.000000" : 31104,
"70.000000" : 31360,
"80.000000" : 31360,
"90.000000" : 43776,
"95.000000" : 51968,
"99.000000" : 62208,
"99.500000" : 66048,
"99.900000" : 132096,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 17322,
"max" : 221956,
"mean" : 35491.405287,
"stddev" : 8595.079541,
"N" : 1567051
},
"bw_min" : 892688,
"bw_max" : 897683,
"bw_agg" : 100.000000,
"bw_mean" : 895548.785714,
"bw_dev" : 1287.687264,
"bw_samples" : 14,
"iops_min" : 223172,
"iops_max" : 224420,
"iops_mean" : 223887.142857,
"iops_stddev" : 321.708625,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 29.585714,
"sys_cpu" : 70.385714,
"ctx" : 13,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.035098,
"50" : 91.088827,
"100" : 8.597908,
"250" : 0.278614,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=89479: Tue Oct 25 23:51:05 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6121MiB/7001msec)
slat (nsec): min=1472, max=25949, avg=1754.16, stdev=224.64
clat (usec): min=15, max=220, avg=33.74, stdev= 8.60
lat (usec): min=17, max=221, avg=35.49, stdev= 8.60
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31],
| 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 44], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 67], 99.90th=[ 133], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=892688, max=897683, per=100.00%, avg=895548.79, stdev=1287.69, samples=14
iops : min=223172, max=224420, avg=223887.14, stdev=321.71, samples=14
lat (usec) : 20=0.04%, 50=91.09%, 100=8.60%, 250=0.28%
cpu : usr=29.59%, sys=70.39%, ctx=13, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567044,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6121MiB (6419MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-xnvme_0.txt"
$ cat /tmp/fio-output_4K_8_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734679,
"timestamp_ms" : 1666734679679,
"time" : "Tue Oct 25 23:51:19 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 6417080320,
"io_kbytes" : 6266680,
"bw_bytes" : 916594817,
"bw" : 895112,
"iops" : 223777.031853,
"runtime" : 7001,
"total_ios" : 1566663,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1312,
"max" : 178064,
"mean" : 2022.793881,
"stddev" : 860.278382,
"N" : 1566663
},
"clat_ns" : {
"min" : 5811,
"max" : 218590,
"mean" : 33429.997546,
"stddev" : 8920.942316,
"N" : 1566670,
"percentile" : {
"1.000000" : 23936,
"5.000000" : 28032,
"10.000000" : 28032,
"20.000000" : 28544,
"30.000000" : 30336,
"40.000000" : 30848,
"50.000000" : 31872,
"60.000000" : 32128,
"70.000000" : 32384,
"80.000000" : 32640,
"90.000000" : 45312,
"95.000000" : 51968,
"99.000000" : 62208,
"99.500000" : 67072,
"99.900000" : 132096,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 15940,
"max" : 221325,
"mean" : 35452.790505,
"stddev" : 8811.650507,
"N" : 1566670
},
"bw_min" : 894184,
"bw_max" : 897432,
"bw_agg" : 100.000000,
"bw_mean" : 895345.000000,
"bw_dev" : 952.534675,
"bw_samples" : 14,
"iops_min" : 223546,
"iops_max" : 224358,
"iops_mean" : 223836.214286,
"iops_stddev" : 238.146748,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 16.414286,
"sys_cpu" : 62.785714,
"ctx" : 539835,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 0.228958,
"50" : 92.130024,
"100" : 7.357294,
"250" : 0.284107,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=89518: Tue Oct 25 23:51:19 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6120MiB/7001msec)
slat (nsec): min=1312, max=178064, avg=2022.79, stdev=860.28
clat (usec): min=5, max=218, avg=33.43, stdev= 8.92
lat (usec): min=15, max=221, avg=35.45, stdev= 8.81
clat percentiles (usec):
| 1.00th=[ 24], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 33],
| 70.00th=[ 33], 80.00th=[ 33], 90.00th=[ 46], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 68], 99.90th=[ 133], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=894184, max=897432, per=100.00%, avg=895345.00, stdev=952.53, samples=14
iops : min=223546, max=224358, avg=223836.21, stdev=238.15, samples=14
lat (usec) : 10=0.01%, 20=0.23%, 50=92.13%, 100=7.36%, 250=0.28%
cpu : usr=16.41%, sys=62.79%, ctx=539835, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1566663,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6120MiB (6417MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-xnvme_1.txt"
$ cat /tmp/fio-output_4K_8_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734693,
"timestamp_ms" : 1666734693305,
"time" : "Tue Oct 25 23:51:33 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 6416519168,
"io_kbytes" : 6266132,
"bw_bytes" : 916514664,
"bw" : 895033,
"iops" : 223757.320383,
"runtime" : 7001,
"total_ios" : 1566525,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1283,
"max" : 37260,
"mean" : 2104.457766,
"stddev" : 885.717940,
"N" : 1566526
},
"clat_ns" : {
"min" : 220,
"max" : 267692,
"mean" : 33332.316245,
"stddev" : 8926.514508,
"N" : 1566532,
"percentile" : {
"1.000000" : 24192,
"5.000000" : 28032,
"10.000000" : 28032,
"20.000000" : 28288,
"30.000000" : 29568,
"40.000000" : 30848,
"50.000000" : 31872,
"60.000000" : 32128,
"70.000000" : 32384,
"80.000000" : 32640,
"90.000000" : 45312,
"95.000000" : 51968,
"99.000000" : 62208,
"99.500000" : 67072,
"99.900000" : 132096,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 15189,
"max" : 269315,
"mean" : 35436.773715,
"stddev" : 8807.724468,
"N" : 1566532
},
"bw_min" : 892808,
"bw_max" : 897400,
"bw_agg" : 100.000000,
"bw_mean" : 895258.142857,
"bw_dev" : 1198.668098,
"bw_samples" : 14,
"iops_min" : 223202,
"iops_max" : 224350,
"iops_mean" : 223814.500000,
"iops_stddev" : 299.664107,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.414286,
"sys_cpu" : 64.100000,
"ctx" : 529590,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 0.256364,
"50" : 92.233191,
"100" : 7.231356,
"250" : 0.279089,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=89557: Tue Oct 25 23:51:33 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6119MiB/7001msec)
slat (nsec): min=1283, max=37260, avg=2104.46, stdev=885.72
clat (nsec): min=220, max=267692, avg=33332.32, stdev=8926.51
lat (usec): min=15, max=269, avg=35.44, stdev= 8.81
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29],
| 30.00th=[ 30], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 33],
| 70.00th=[ 33], 80.00th=[ 33], 90.00th=[ 46], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 68], 99.90th=[ 133], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=892808, max=897400, per=100.00%, avg=895258.14, stdev=1198.67, samples=14
iops : min=223202, max=224350, avg=223814.50, stdev=299.66, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=0.26%, 50=92.23%, 100=7.23%
lat (usec) : 250=0.28%, 500=0.01%
cpu : usr=15.41%, sys=64.10%, ctx=529590, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1566525,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6119MiB (6417MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-xnvme_2.txt"
$ cat /tmp/fio-output_4K_1_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733157,
"timestamp_ms" : 1666733157028,
"time" : "Tue Oct 25 23:25:57 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 1804107776,
"io_kbytes" : 1761824,
"bw_bytes" : 257729682,
"bw" : 251689,
"iops" : 62922.285714,
"runtime" : 7000,
"total_ios" : 440456,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2465,
"max" : 34736,
"mean" : 2646.838867,
"stddev" : 213.387163,
"N" : 440456
},
"clat_ns" : {
"min" : 240,
"max" : 105898,
"mean" : 12567.897549,
"stddev" : 1992.233414,
"N" : 440456,
"percentile" : {
"1.000000" : 12096,
"5.000000" : 12224,
"10.000000" : 12224,
"20.000000" : 12224,
"30.000000" : 12224,
"40.000000" : 12224,
"50.000000" : 12352,
"60.000000" : 12352,
"70.000000" : 12352,
"80.000000" : 12352,
"90.000000" : 12352,
"95.000000" : 12608,
"99.000000" : 22400,
"99.500000" : 27520,
"99.900000" : 36608,
"99.950000" : 49920,
"99.990000" : 67072
}
},
"lat_ns" : {
"min" : 11952,
"max" : 108593,
"mean" : 15214.736416,
"stddev" : 2006.232859,
"N" : 440456
},
"bw_min" : 250789,
"bw_max" : 252440,
"bw_agg" : 100.000000,
"bw_mean" : 251748.785714,
"bw_dev" : 511.742450,
"bw_samples" : 14,
"iops_min" : 62697,
"iops_max" : 63110,
"iops_mean" : 62937.142857,
"iops_stddev" : 127.995106,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 9.485714,
"sys_cpu" : 38.314286,
"ctx" : 440441,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 98.659571,
"50" : 1.284351,
"100" : 0.047905,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 623550,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 5990,
"write_ticks" : 0,
"in_queue" : 5990,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=84848: Tue Oct 25 23:25:57 2022
read: IOPS=62.9k, BW=246MiB/s (258MB/s)(1721MiB/7000msec)
slat (nsec): min=2465, max=34736, avg=2646.84, stdev=213.39
clat (nsec): min=240, max=105898, avg=12567.90, stdev=1992.23
lat (usec): min=11, max=108, avg=15.21, stdev= 2.01
clat percentiles (nsec):
| 1.00th=[12096], 5.00th=[12224], 10.00th=[12224], 20.00th=[12224],
| 30.00th=[12224], 40.00th=[12224], 50.00th=[12352], 60.00th=[12352],
| 70.00th=[12352], 80.00th=[12352], 90.00th=[12352], 95.00th=[12608],
| 99.00th=[22400], 99.50th=[27520], 99.90th=[36608], 99.95th=[49920],
| 99.99th=[67072]
bw ( KiB/s): min=250789, max=252440, per=100.00%, avg=251748.79, stdev=511.74, samples=14
iops : min=62697, max=63110, avg=62937.14, stdev=128.00, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=98.66%, 50=1.28%, 100=0.05%
lat (usec) : 250=0.01%
cpu : usr=9.49%, sys=38.31%, ctx=440441, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=440456,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=246MiB/s (258MB/s), 246MiB/s-246MiB/s (258MB/s-258MB/s), io=1721MiB (1804MB), run=7000-7000msec
Disk stats (read/write):
nvme0n1: ios=623550/0, merge=0/0, ticks=5990/0, in_queue=5990, util=99.09%
$ cat /tmp/fio-output_4K_8_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734706,
"timestamp_ms" : 1666734706937,
"time" : "Tue Oct 25 23:51:46 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 6417715200,
"io_kbytes" : 6267300,
"bw_bytes" : 916685502,
"bw" : 895200,
"iops" : 223799.028710,
"runtime" : 7001,
"total_ios" : 1566817,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1292,
"max" : 57117,
"mean" : 2005.485569,
"stddev" : 861.647587,
"N" : 1566818
},
"clat_ns" : {
"min" : 261,
"max" : 206247,
"mean" : 33462.330400,
"stddev" : 8878.873955,
"N" : 1566824,
"percentile" : {
"1.000000" : 23936,
"5.000000" : 28032,
"10.000000" : 28032,
"20.000000" : 28544,
"30.000000" : 30336,
"40.000000" : 30848,
"50.000000" : 31872,
"60.000000" : 32128,
"70.000000" : 32384,
"80.000000" : 32640,
"90.000000" : 45312,
"95.000000" : 51968,
"99.000000" : 62720,
"99.500000" : 67072,
"99.900000" : 130560,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 16311,
"max" : 207599,
"mean" : 35467.815561,
"stddev" : 8771.771322,
"N" : 1566824
},
"bw_min" : 893368,
"bw_max" : 897939,
"bw_agg" : 100.000000,
"bw_mean" : 895407.071429,
"bw_dev" : 1490.748416,
"bw_samples" : 14,
"iops_min" : 223342,
"iops_max" : 224484,
"iops_mean" : 223851.714286,
"iops_stddev" : 372.323149,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.357143,
"sys_cpu" : 63.742857,
"ctx" : 541721,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 0.272080,
"50" : 92.033020,
"100" : 7.417778,
"250" : 0.277314,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=89597: Tue Oct 25 23:51:46 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6120MiB/7001msec)
slat (nsec): min=1292, max=57117, avg=2005.49, stdev=861.65
clat (nsec): min=261, max=206247, avg=33462.33, stdev=8878.87
lat (usec): min=16, max=207, avg=35.47, stdev= 8.77
clat percentiles (usec):
| 1.00th=[ 24], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 33],
| 70.00th=[ 33], 80.00th=[ 33], 90.00th=[ 46], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 68], 99.90th=[ 131], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=893368, max=897939, per=100.00%, avg=895407.07, stdev=1490.75, samples=14
iops : min=223342, max=224484, avg=223851.71, stdev=372.32, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=0.27%, 50=92.03%, 100=7.42%
lat (usec) : 250=0.28%
cpu : usr=15.36%, sys=63.74%, ctx=541721, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1566817,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6120MiB (6418MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-reference_0.txt"
$ cat /tmp/fio-output_4K_8_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734720,
"timestamp_ms" : 1666734720577,
"time" : "Tue Oct 25 23:52:00 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 6421184512,
"io_kbytes" : 6270688,
"bw_bytes" : 917181047,
"bw" : 895684,
"iops" : 223920.011427,
"runtime" : 7001,
"total_ios" : 1567664,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1372,
"max" : 23945,
"mean" : 2044.074586,
"stddev" : 778.052273,
"N" : 1567665
},
"clat_ns" : {
"min" : 15299,
"max" : 193182,
"mean" : 33405.325188,
"stddev" : 8836.893542,
"N" : 1567671,
"percentile" : {
"1.000000" : 24448,
"5.000000" : 28288,
"10.000000" : 28288,
"20.000000" : 28800,
"30.000000" : 29312,
"40.000000" : 30592,
"50.000000" : 31616,
"60.000000" : 32128,
"70.000000" : 32384,
"80.000000" : 33536,
"90.000000" : 44800,
"95.000000" : 51968,
"99.000000" : 62720,
"99.500000" : 67072,
"99.900000" : 128512,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 17783,
"max" : 194615,
"mean" : 35449.399103,
"stddev" : 8754.063693,
"N" : 1567671
},
"bw_min" : 894608,
"bw_max" : 897456,
"bw_agg" : 100.000000,
"bw_mean" : 896045.714286,
"bw_dev" : 800.433289,
"bw_samples" : 14,
"iops_min" : 223652,
"iops_max" : 224364,
"iops_mean" : 224011.428571,
"iops_stddev" : 200.229758,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 21.657143,
"sys_cpu" : 57.442857,
"ctx" : 533446,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999936,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.044142,
"50" : 92.606005,
"100" : 7.090231,
"250" : 0.260068,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2235295,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 71869,
"write_ticks" : 0,
"in_queue" : 71869,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=89636: Tue Oct 25 23:52:00 2022
read: IOPS=224k, BW=875MiB/s (917MB/s)(6124MiB/7001msec)
slat (nsec): min=1372, max=23945, avg=2044.07, stdev=778.05
clat (usec): min=15, max=193, avg=33.41, stdev= 8.84
lat (usec): min=17, max=194, avg=35.45, stdev= 8.75
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29],
| 30.00th=[ 30], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 33],
| 70.00th=[ 33], 80.00th=[ 34], 90.00th=[ 45], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 68], 99.90th=[ 129], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=894608, max=897456, per=100.00%, avg=896045.71, stdev=800.43, samples=14
iops : min=223652, max=224364, avg=224011.43, stdev=200.23, samples=14
lat (usec) : 20=0.04%, 50=92.61%, 100=7.09%, 250=0.26%
cpu : usr=21.66%, sys=57.44%, ctx=533446, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567664,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6124MiB (6421MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2235295/0, merge=0/0, ticks=71869/0, in_queue=71869, util=99.09%
$ rm /tmp/fio-output_4K_8_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-reference_1.txt"
$ cat /tmp/fio-output_4K_8_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734734,
"timestamp_ms" : 1666734734194,
"time" : "Tue Oct 25 23:52:14 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 6419886080,
"io_kbytes" : 6269420,
"bw_bytes" : 916995583,
"bw" : 895503,
"iops" : 223874.875018,
"runtime" : 7001,
"total_ios" : 1567348,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1422,
"max" : 42911,
"mean" : 2070.390944,
"stddev" : 758.523721,
"N" : 1567348
},
"clat_ns" : {
"min" : 391,
"max" : 177383,
"mean" : 33384.698160,
"stddev" : 8925.881402,
"N" : 1567355,
"percentile" : {
"1.000000" : 22912,
"5.000000" : 27264,
"10.000000" : 28544,
"20.000000" : 28800,
"30.000000" : 29056,
"40.000000" : 29568,
"50.000000" : 30848,
"60.000000" : 32128,
"70.000000" : 32640,
"80.000000" : 34048,
"90.000000" : 44800,
"95.000000" : 51968,
"99.000000" : 62720,
"99.500000" : 67072,
"99.900000" : 128512,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 14407,
"max" : 178875,
"mean" : 35455.088327,
"stddev" : 8862.894253,
"N" : 1567355
},
"bw_min" : 893552,
"bw_max" : 897856,
"bw_agg" : 100.000000,
"bw_mean" : 895919.923077,
"bw_dev" : 1175.135344,
"bw_samples" : 13,
"iops_min" : 223388,
"iops_max" : 224464,
"iops_mean" : 223979.923077,
"iops_stddev" : 293.956477,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 25.357143,
"sys_cpu" : 52.514286,
"ctx" : 563635,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999936,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.000000,
"20" : 0.058570,
"50" : 92.628185,
"100" : 7.051848,
"250" : 0.260950,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2177732,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 69367,
"write_ticks" : 0,
"in_queue" : 69367,
"util" : 99.085923
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=89676: Tue Oct 25 23:52:14 2022
read: IOPS=224k, BW=875MiB/s (917MB/s)(6122MiB/7001msec)
slat (nsec): min=1422, max=42911, avg=2070.39, stdev=758.52
clat (nsec): min=391, max=177383, avg=33384.70, stdev=8925.88
lat (usec): min=14, max=178, avg=35.46, stdev= 8.86
clat percentiles (usec):
| 1.00th=[ 23], 5.00th=[ 28], 10.00th=[ 29], 20.00th=[ 29],
| 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 31], 60.00th=[ 33],
| 70.00th=[ 33], 80.00th=[ 35], 90.00th=[ 45], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 68], 99.90th=[ 129], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=893552, max=897856, per=100.00%, avg=895919.92, stdev=1175.14, samples=13
iops : min=223388, max=224464, avg=223979.92, stdev=293.96, samples=13
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 20=0.06%, 50=92.63%, 100=7.05%, 250=0.26%
cpu : usr=25.36%, sys=52.51%, ctx=563635, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567348,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6122MiB (6420MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2177732/0, merge=0/0, ticks=69367/0, in_queue=69367, util=99.09%
$ rm /tmp/fio-output_4K_8_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-reference_2.txt"
$ cat /tmp/fio-output_4K_8_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734747,
"timestamp_ms" : 1666734747818,
"time" : "Tue Oct 25 23:52:27 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 6419091456,
"io_kbytes" : 6268644,
"bw_bytes" : 916882081,
"bw" : 895392,
"iops" : 223847.164691,
"runtime" : 7001,
"total_ios" : 1567154,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1332,
"max" : 1297263,
"mean" : 2019.222772,
"stddev" : 1292.972535,
"N" : 1567154
},
"clat_ns" : {
"min" : 12954,
"max" : 1329383,
"mean" : 33446.079906,
"stddev" : 9235.556673,
"N" : 1567161,
"percentile" : {
"1.000000" : 24192,
"5.000000" : 28032,
"10.000000" : 28288,
"20.000000" : 28800,
"30.000000" : 29568,
"40.000000" : 30592,
"50.000000" : 31616,
"60.000000" : 32128,
"70.000000" : 32384,
"80.000000" : 33536,
"90.000000" : 44800,
"95.000000" : 51968,
"99.000000" : 62720,
"99.500000" : 67072,
"99.900000" : 129536,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 17462,
"max" : 1332139,
"mean" : 35465.301490,
"stddev" : 9217.281644,
"N" : 1567161
},
"bw_min" : 892584,
"bw_max" : 897699,
"bw_agg" : 100.000000,
"bw_mean" : 895743.142857,
"bw_dev" : 1372.428272,
"bw_samples" : 14,
"iops_min" : 223146,
"iops_max" : 224424,
"iops_mean" : 223935.714286,
"iops_stddev" : 343.066584,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 22.885714,
"sys_cpu" : 55.914286,
"ctx" : 540455,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999936,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.049261,
"50" : 92.557081,
"100" : 7.134079,
"250" : 0.259579,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2234710,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 71868,
"write_ticks" : 0,
"in_queue" : 71868,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=89716: Tue Oct 25 23:52:27 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6122MiB/7001msec)
slat (nsec): min=1332, max=1297.3k, avg=2019.22, stdev=1292.97
clat (usec): min=12, max=1329, avg=33.45, stdev= 9.24
lat (usec): min=17, max=1332, avg=35.47, stdev= 9.22
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29],
| 30.00th=[ 30], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 33],
| 70.00th=[ 33], 80.00th=[ 34], 90.00th=[ 45], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 68], 99.90th=[ 130], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=892584, max=897699, per=100.00%, avg=895743.14, stdev=1372.43, samples=14
iops : min=223146, max=224424, avg=223935.71, stdev=343.07, samples=14
lat (usec) : 20=0.05%, 50=92.56%, 100=7.13%, 250=0.26%
lat (msec) : 2=0.01%
cpu : usr=22.89%, sys=55.91%, ctx=540455, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567154,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6122MiB (6419MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2234710/0, merge=0/0, ticks=71868/0, in_queue=71868, util=99.09%
$ rm /tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt': No such file or directory
$ rm /tmp/fio-output_4K_8_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-bdev_aio_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-bdev_aio_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734762,
"timestamp_ms" : 1666734762792,
"time" : "Tue Oct 25 23:52:42 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6415536128,
"io_kbytes" : 6265172,
"bw_bytes" : 916374250,
"bw" : 894896,
"iops" : 223723.039566,
"runtime" : 7001,
"total_ios" : 1566285,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1553,
"max" : 184837,
"mean" : 1667.843817,
"stddev" : 320.534687,
"N" : 1566286
},
"clat_ns" : {
"min" : 541,
"max" : 215374,
"mean" : 33789.432111,
"stddev" : 8671.015722,
"N" : 1566285,
"percentile" : {
"1.000000" : 24448,
"5.000000" : 30848,
"10.000000" : 30848,
"20.000000" : 31104,
"30.000000" : 31104,
"40.000000" : 31104,
"50.000000" : 31360,
"60.000000" : 31360,
"70.000000" : 31360,
"80.000000" : 31360,
"90.000000" : 43776,
"95.000000" : 51968,
"99.000000" : 62208,
"99.500000" : 66048,
"99.900000" : 134144,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 14217,
"max" : 217007,
"mean" : 35457.275931,
"stddev" : 8658.022173,
"N" : 1566285
},
"bw_min" : 893960,
"bw_max" : 896200,
"bw_agg" : 100.000000,
"bw_mean" : 895105.071429,
"bw_dev" : 806.860627,
"bw_samples" : 14,
"iops_min" : 223490,
"iops_max" : 224050,
"iops_mean" : 223776.214286,
"iops_stddev" : 201.682225,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 67.585714,
"sys_cpu" : 32.385714,
"ctx" : 14,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.100000,
"4" : 1.306531,
"8" : 98.689702,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.996755,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 0.047437,
"50" : 90.986251,
"100" : 8.674794,
"250" : 0.291326,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=89765: Tue Oct 25 23:52:42 2022
read: IOPS=224k, BW=874MiB/s (916MB/s)(6118MiB/7001msec)
slat (nsec): min=1553, max=184837, avg=1667.84, stdev=320.53
clat (nsec): min=541, max=215374, avg=33789.43, stdev=8671.02
lat (usec): min=14, max=217, avg=35.46, stdev= 8.66
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32],
| 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 44], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 67], 99.90th=[ 135], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=893960, max=896200, per=100.00%, avg=895105.07, stdev=806.86, samples=14
iops : min=223490, max=224050, avg=223776.21, stdev=201.68, samples=14
lat (nsec) : 750=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=0.05%, 50=90.99%, 100=8.67%
lat (usec) : 250=0.29%
cpu : usr=67.59%, sys=32.39%, ctx=14, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=1.3%, 8=98.7%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1566285,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (916MB/s), 874MiB/s-874MiB/s (916MB/s-916MB/s), io=6118MiB (6416MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-bdev_aio_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-bdev_aio_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734777,
"timestamp_ms" : 1666734777907,
"time" : "Tue Oct 25 23:52:57 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6419636224,
"io_kbytes" : 6269176,
"bw_bytes" : 916959894,
"bw" : 895468,
"iops" : 223866.161977,
"runtime" : 7001,
"total_ios" : 1567287,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1562,
"max" : 23695,
"mean" : 1655.664724,
"stddev" : 232.107596,
"N" : 1567287
},
"clat_ns" : {
"min" : 7424,
"max" : 174908,
"mean" : 33773.507443,
"stddev" : 8548.878283,
"N" : 1567294,
"percentile" : {
"1.000000" : 24704,
"5.000000" : 30848,
"10.000000" : 31104,
"20.000000" : 31104,
"30.000000" : 31104,
"40.000000" : 31104,
"50.000000" : 31360,
"60.000000" : 31360,
"70.000000" : 31360,
"80.000000" : 31360,
"90.000000" : 43776,
"95.000000" : 51968,
"99.000000" : 62208,
"99.500000" : 66048,
"99.900000" : 130560,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 14487,
"max" : 176551,
"mean" : 35429.171972,
"stddev" : 8534.978594,
"N" : 1567294
},
"bw_min" : 892608,
"bw_max" : 898528,
"bw_agg" : 100.000000,
"bw_mean" : 895684.785714,
"bw_dev" : 1449.683476,
"bw_samples" : 14,
"iops_min" : 223152,
"iops_max" : 224632,
"iops_mean" : 223921.142857,
"iops_stddev" : 362.336617,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 68.328571,
"sys_cpu" : 31.642857,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.100000,
"4" : 1.275835,
"8" : 98.722952,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.997849,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 0.037517,
"50" : 91.020981,
"100" : 8.667525,
"250" : 0.274359,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=89814: Tue Oct 25 23:52:57 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6122MiB/7001msec)
slat (nsec): min=1562, max=23695, avg=1655.66, stdev=232.11
clat (usec): min=7, max=174, avg=33.77, stdev= 8.55
lat (usec): min=14, max=176, avg=35.43, stdev= 8.53
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 32], 20.00th=[ 32],
| 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 44], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 67], 99.90th=[ 131], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=892608, max=898528, per=100.00%, avg=895684.79, stdev=1449.68, samples=14
iops : min=223152, max=224632, avg=223921.14, stdev=362.34, samples=14
lat (usec) : 10=0.01%, 20=0.04%, 50=91.02%, 100=8.67%, 250=0.27%
cpu : usr=68.33%, sys=31.64%, ctx=10, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=1.3%, 8=98.7%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567287,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6122MiB (6420MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-bdev_aio_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-bdev_aio_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734793,
"timestamp_ms" : 1666734793038,
"time" : "Tue Oct 25 23:53:13 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6418944000,
"io_kbytes" : 6268500,
"bw_bytes" : 916861019,
"bw" : 895372,
"iops" : 223841.879731,
"runtime" : 7001,
"total_ios" : 1567117,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1542,
"max" : 51917,
"mean" : 1638.791215,
"stddev" : 252.351930,
"N" : 1567118
},
"clat_ns" : {
"min" : 12954,
"max" : 173766,
"mean" : 33817.441936,
"stddev" : 8542.652012,
"N" : 1567117,
"percentile" : {
"1.000000" : 24448,
"5.000000" : 30848,
"10.000000" : 31104,
"20.000000" : 31104,
"30.000000" : 31104,
"40.000000" : 31104,
"50.000000" : 31360,
"60.000000" : 31360,
"70.000000" : 31360,
"80.000000" : 31616,
"90.000000" : 43776,
"95.000000" : 51968,
"99.000000" : 62208,
"99.500000" : 66048,
"99.900000" : 129536,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 15129,
"max" : 175399,
"mean" : 35456.233161,
"stddev" : 8527.216217,
"N" : 1567117
},
"bw_min" : 894312,
"bw_max" : 896672,
"bw_agg" : 100.000000,
"bw_mean" : 895574.142857,
"bw_dev" : 761.594467,
"bw_samples" : 14,
"iops_min" : 223578,
"iops_max" : 224168,
"iops_mean" : 223893.500000,
"iops_stddev" : 190.409579,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 71.014286,
"sys_cpu" : 28.971429,
"ctx" : 46,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.100000,
"4" : 1.324087,
"8" : 98.671446,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.995953,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.057941,
"50" : 90.970872,
"100" : 8.694820,
"250" : 0.276367,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=89863: Tue Oct 25 23:53:13 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6122MiB/7001msec)
slat (nsec): min=1542, max=51917, avg=1638.79, stdev=252.35
clat (usec): min=12, max=173, avg=33.82, stdev= 8.54
lat (usec): min=15, max=175, avg=35.46, stdev= 8.53
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 32], 20.00th=[ 32],
| 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 44], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 67], 99.90th=[ 130], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=894312, max=896672, per=100.00%, avg=895574.14, stdev=761.59, samples=14
iops : min=223578, max=224168, avg=223893.50, stdev=190.41, samples=14
lat (usec) : 20=0.06%, 50=90.97%, 100=8.69%, 250=0.28%
cpu : usr=71.01%, sys=28.97%, ctx=46, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=1.3%, 8=98.7%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567117,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6122MiB (6419MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734808,
"timestamp_ms" : 1666734808232,
"time" : "Tue Oct 25 23:53:28 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6424309760,
"io_kbytes" : 6273740,
"bw_bytes" : 917627447,
"bw" : 896120,
"iops" : 224028.995858,
"runtime" : 7001,
"total_ios" : 1568427,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1683,
"max" : 37610,
"mean" : 1920.989116,
"stddev" : 147.325480,
"N" : 1568428
},
"clat_ns" : {
"min" : 1113,
"max" : 185858,
"mean" : 33505.838697,
"stddev" : 8743.561843,
"N" : 1568430,
"percentile" : {
"1.000000" : 23168,
"5.000000" : 28288,
"10.000000" : 30336,
"20.000000" : 30592,
"30.000000" : 30848,
"40.000000" : 30848,
"50.000000" : 30848,
"60.000000" : 31104,
"70.000000" : 31104,
"80.000000" : 31360,
"90.000000" : 44800,
"95.000000" : 51968,
"99.000000" : 62208,
"99.500000" : 67072,
"99.900000" : 130560,
"99.950000" : 142336,
"99.990000" : 160768
}
},
"lat_ns" : {
"min" : 13776,
"max" : 187792,
"mean" : 35426.827932,
"stddev" : 8743.318794,
"N" : 1568430
},
"bw_min" : 894136,
"bw_max" : 898096,
"bw_agg" : 100.000000,
"bw_mean" : 896613.785714,
"bw_dev" : 1166.083001,
"bw_samples" : 14,
"iops_min" : 223534,
"iops_max" : 224524,
"iops_mean" : 224153.357143,
"iops_stddev" : 291.452492,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 41.528571,
"sys_cpu" : 58.442857,
"ctx" : 18,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.100000,
"4" : 0.739403,
"8" : 99.259832,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999232,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.127963,
"50" : 91.673441,
"100" : 7.954977,
"250" : 0.243747,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=89912: Tue Oct 25 23:53:28 2022
read: IOPS=224k, BW=875MiB/s (918MB/s)(6127MiB/7001msec)
slat (nsec): min=1683, max=37610, avg=1920.99, stdev=147.33
clat (nsec): min=1113, max=185858, avg=33505.84, stdev=8743.56
lat (usec): min=13, max=187, avg=35.43, stdev= 8.74
clat percentiles (usec):
| 1.00th=[ 24], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 45], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 68], 99.90th=[ 131], 99.95th=[ 143],
| 99.99th=[ 161]
bw ( KiB/s): min=894136, max=898096, per=100.00%, avg=896613.79, stdev=1166.08, samples=14
iops : min=223534, max=224524, avg=224153.36, stdev=291.45, samples=14
lat (usec) : 2=0.01%, 20=0.13%, 50=91.67%, 100=7.95%, 250=0.24%
cpu : usr=41.53%, sys=58.44%, ctx=18, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.7%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1568427,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=875MiB/s (918MB/s), 875MiB/s-875MiB/s (918MB/s-918MB/s), io=6127MiB (6424MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734823,
"timestamp_ms" : 1666734823417,
"time" : "Tue Oct 25 23:53:43 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6421917696,
"io_kbytes" : 6271404,
"bw_bytes" : 917285772,
"bw" : 895786,
"iops" : 223945.722040,
"runtime" : 7001,
"total_ios" : 1567844,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1623,
"max" : 74419,
"mean" : 1755.833047,
"stddev" : 331.760129,
"N" : 1567844
},
"clat_ns" : {
"min" : 1042,
"max" : 177283,
"mean" : 33637.009474,
"stddev" : 8459.789321,
"N" : 1567851,
"percentile" : {
"1.000000" : 24704,
"5.000000" : 30592,
"10.000000" : 30848,
"20.000000" : 30848,
"30.000000" : 31104,
"40.000000" : 31104,
"50.000000" : 31104,
"60.000000" : 31360,
"70.000000" : 31360,
"80.000000" : 31616,
"90.000000" : 43776,
"95.000000" : 51968,
"99.000000" : 62208,
"99.500000" : 64768,
"99.900000" : 127488,
"99.950000" : 138240,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 13115,
"max" : 178976,
"mean" : 35392.842113,
"stddev" : 8437.240273,
"N" : 1567851
},
"bw_min" : 894312,
"bw_max" : 897768,
"bw_agg" : 100.000000,
"bw_mean" : 896021.142857,
"bw_dev" : 1070.104442,
"bw_samples" : 14,
"iops_min" : 223578,
"iops_max" : 224442,
"iops_mean" : 224005.285714,
"iops_stddev" : 267.558312,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 48.500000,
"sys_cpu" : 51.471429,
"ctx" : 25,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.100000,
"4" : 1.606537,
"8" : 98.385299,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.995820,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 0.055618,
"50" : 91.209266,
"100" : 8.477757,
"250" : 0.257360,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=89963: Tue Oct 25 23:53:43 2022
read: IOPS=224k, BW=875MiB/s (917MB/s)(6124MiB/7001msec)
slat (nsec): min=1623, max=74419, avg=1755.83, stdev=331.76
clat (nsec): min=1042, max=177283, avg=33637.01, stdev=8459.79
lat (usec): min=13, max=178, avg=35.39, stdev= 8.44
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31],
| 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 44], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 65], 99.90th=[ 128], 99.95th=[ 139],
| 99.99th=[ 157]
bw ( KiB/s): min=894312, max=897768, per=100.00%, avg=896021.14, stdev=1070.10, samples=14
iops : min=223578, max=224442, avg=224005.29, stdev=267.56, samples=14
lat (usec) : 2=0.01%, 10=0.01%, 20=0.06%, 50=91.21%, 100=8.48%
lat (usec) : 250=0.26%
cpu : usr=48.50%, sys=51.47%, ctx=25, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=1.6%, 8=98.4%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567844,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6124MiB (6422MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734838,
"timestamp_ms" : 1666734838539,
"time" : "Tue Oct 25 23:53:58 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 6419918848,
"io_kbytes" : 6269452,
"bw_bytes" : 917000263,
"bw" : 895508,
"iops" : 223876.017712,
"runtime" : 7001,
"total_ios" : 1567356,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1583,
"max" : 180989,
"mean" : 1700.893440,
"stddev" : 355.436322,
"N" : 1567356
},
"clat_ns" : {
"min" : 1022,
"max" : 221516,
"mean" : 33718.714563,
"stddev" : 8534.695940,
"N" : 1567363,
"percentile" : {
"1.000000" : 24448,
"5.000000" : 30592,
"10.000000" : 30848,
"20.000000" : 31104,
"30.000000" : 31104,
"40.000000" : 31104,
"50.000000" : 31104,
"60.000000" : 31360,
"70.000000" : 31360,
"80.000000" : 31616,
"90.000000" : 43776,
"95.000000" : 51968,
"99.000000" : 62208,
"99.500000" : 65280,
"99.900000" : 129536,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 15730,
"max" : 223139,
"mean" : 35419.607731,
"stddev" : 8515.962916,
"N" : 1567363
},
"bw_min" : 894160,
"bw_max" : 898496,
"bw_agg" : 100.000000,
"bw_mean" : 895716.928571,
"bw_dev" : 996.908025,
"bw_samples" : 14,
"iops_min" : 223540,
"iops_max" : 224624,
"iops_mean" : 223929.357143,
"iops_stddev" : 249.118324,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 48.785714,
"sys_cpu" : 51.185714,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.100000,
"4" : 1.516120,
"8" : 98.479159,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.995950,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 0.054040,
"50" : 91.150256,
"100" : 8.527929,
"250" : 0.267967,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=90012: Tue Oct 25 23:53:58 2022
read: IOPS=224k, BW=875MiB/s (917MB/s)(6123MiB/7001msec)
slat (nsec): min=1583, max=180989, avg=1700.89, stdev=355.44
clat (nsec): min=1022, max=221516, avg=33718.71, stdev=8534.70
lat (usec): min=15, max=223, avg=35.42, stdev= 8.52
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32],
| 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 44], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 66], 99.90th=[ 130], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=894160, max=898496, per=100.00%, avg=895716.93, stdev=996.91, samples=14
iops : min=223540, max=224624, avg=223929.36, stdev=249.12, samples=14
lat (usec) : 2=0.01%, 10=0.01%, 20=0.05%, 50=91.15%, 100=8.53%
lat (usec) : 250=0.27%
cpu : usr=48.79%, sys=51.19%, ctx=10, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=1.5%, 8=98.5%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1567356,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6123MiB (6420MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-xnvme_0.txt"
$ cat /tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733172,
"timestamp_ms" : 1666733172034,
"time" : "Tue Oct 25 23:26:12 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2409836544,
"io_kbytes" : 2353356,
"bw_bytes" : 344213190,
"bw" : 336145,
"iops" : 84036.423368,
"runtime" : 7001,
"total_ios" : 588339,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 160,
"max" : 10810,
"mean" : 173.542789,
"stddev" : 22.695499,
"N" : 588339
},
"clat_ns" : {
"min" : 10410,
"max" : 78336,
"mean" : 11484.418524,
"stddev" : 2190.358012,
"N" : 588339,
"percentile" : {
"1.000000" : 10944,
"5.000000" : 10944,
"10.000000" : 11072,
"20.000000" : 11072,
"30.000000" : 11072,
"40.000000" : 11072,
"50.000000" : 11072,
"60.000000" : 11200,
"70.000000" : 11200,
"80.000000" : 11328,
"90.000000" : 11328,
"95.000000" : 11968,
"99.000000" : 22144,
"99.500000" : 28288,
"99.900000" : 28544,
"99.950000" : 43776,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 10580,
"max" : 78517,
"mean" : 11657.961313,
"stddev" : 2190.707867,
"N" : 588339
},
"bw_min" : 335000,
"bw_max" : 337450,
"bw_agg" : 100.000000,
"bw_mean" : 336330.692308,
"bw_dev" : 777.678960,
"bw_samples" : 13,
"iops_min" : 83750,
"iops_max" : 84362,
"iops_mean" : 84082.615385,
"iops_stddev" : 194.345714,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 83.014286,
"sys_cpu" : 16.971429,
"ctx" : 16,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 98.617124,
"50" : 1.344293,
"100" : 0.038583,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=84897: Tue Oct 25 23:26:12 2022
read: IOPS=84.0k, BW=328MiB/s (344MB/s)(2298MiB/7001msec)
slat (nsec): min=160, max=10810, avg=173.54, stdev=22.70
clat (nsec): min=10410, max=78336, avg=11484.42, stdev=2190.36
lat (nsec): min=10580, max=78517, avg=11657.96, stdev=2190.71
clat percentiles (nsec):
| 1.00th=[10944], 5.00th=[10944], 10.00th=[11072], 20.00th=[11072],
| 30.00th=[11072], 40.00th=[11072], 50.00th=[11072], 60.00th=[11200],
| 70.00th=[11200], 80.00th=[11328], 90.00th=[11328], 95.00th=[11968],
| 99.00th=[22144], 99.50th=[28288], 99.90th=[28544], 99.95th=[43776],
| 99.99th=[68096]
bw ( KiB/s): min=335000, max=337450, per=100.00%, avg=336330.69, stdev=777.68, samples=13
iops : min=83750, max=84362, avg=84082.62, stdev=194.35, samples=13
lat (usec) : 20=98.62%, 50=1.34%, 100=0.04%
cpu : usr=83.01%, sys=16.97%, ctx=16, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=588339,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=2298MiB (2410MB), run=7001-7001msec
$ cat /tmp/fio-output_4K_8_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734852,
"timestamp_ms" : 1666734852329,
"time" : "Tue Oct 25 23:54:12 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 6413660160,
"io_kbytes" : 6263340,
"bw_bytes" : 916106293,
"bw" : 894635,
"iops" : 223657.620340,
"runtime" : 7001,
"total_ios" : 1565827,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1463,
"max" : 42490,
"mean" : 1879.747679,
"stddev" : 476.780147,
"N" : 1565828
},
"clat_ns" : {
"min" : 2766,
"max" : 250209,
"mean" : 33625.378656,
"stddev" : 9412.417229,
"N" : 1565834,
"percentile" : {
"1.000000" : 22912,
"5.000000" : 25984,
"10.000000" : 27264,
"20.000000" : 28544,
"30.000000" : 29312,
"40.000000" : 30336,
"50.000000" : 31104,
"60.000000" : 31872,
"70.000000" : 33024,
"80.000000" : 36608,
"90.000000" : 45312,
"95.000000" : 52480,
"99.000000" : 63232,
"99.500000" : 69120,
"99.900000" : 134144,
"99.950000" : 142336,
"99.990000" : 160768
}
},
"lat_ns" : {
"min" : 15689,
"max" : 262933,
"mean" : 35505.126049,
"stddev" : 9396.777144,
"N" : 1565834
},
"bw_min" : 887864,
"bw_max" : 897378,
"bw_agg" : 100.000000,
"bw_mean" : 895042.928571,
"bw_dev" : 2300.390016,
"bw_samples" : 14,
"iops_min" : 221966,
"iops_max" : 224344,
"iops_mean" : 223760.500000,
"iops_stddev" : 574.996020,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 22.085714,
"sys_cpu" : 54.371429,
"ctx" : 539751,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.000000,
"20" : 0.414669,
"50" : 92.437734,
"100" : 6.875664,
"250" : 0.272188,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=90051: Tue Oct 25 23:54:12 2022
read: IOPS=224k, BW=874MiB/s (916MB/s)(6117MiB/7001msec)
slat (nsec): min=1463, max=42490, avg=1879.75, stdev=476.78
clat (usec): min=2, max=250, avg=33.63, stdev= 9.41
lat (usec): min=15, max=262, avg=35.51, stdev= 9.40
clat percentiles (usec):
| 1.00th=[ 23], 5.00th=[ 26], 10.00th=[ 28], 20.00th=[ 29],
| 30.00th=[ 30], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 34], 80.00th=[ 37], 90.00th=[ 46], 95.00th=[ 53],
| 99.00th=[ 64], 99.50th=[ 70], 99.90th=[ 135], 99.95th=[ 143],
| 99.99th=[ 161]
bw ( KiB/s): min=887864, max=897378, per=100.00%, avg=895042.93, stdev=2300.39, samples=14
iops : min=221966, max=224344, avg=223760.50, stdev=575.00, samples=14
lat (usec) : 4=0.01%, 20=0.41%, 50=92.44%, 100=6.88%, 250=0.27%
lat (usec) : 500=0.01%
cpu : usr=22.09%, sys=54.37%, ctx=539751, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1565827,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (916MB/s), 874MiB/s-874MiB/s (916MB/s-916MB/s), io=6117MiB (6414MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-xnvme_1.txt"
$ cat /tmp/fio-output_4K_8_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734866,
"timestamp_ms" : 1666734866018,
"time" : "Tue Oct 25 23:54:26 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 6422921216,
"io_kbytes" : 6272384,
"bw_bytes" : 917429112,
"bw" : 895926,
"iops" : 223980.717040,
"runtime" : 7001,
"total_ios" : 1568089,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1482,
"max" : 25568,
"mean" : 1774.253431,
"stddev" : 161.376328,
"N" : 1568089
},
"clat_ns" : {
"min" : 13826,
"max" : 201578,
"mean" : 33675.984157,
"stddev" : 9110.646423,
"N" : 1568096,
"percentile" : {
"1.000000" : 23936,
"5.000000" : 25984,
"10.000000" : 27264,
"20.000000" : 29056,
"30.000000" : 30080,
"40.000000" : 30848,
"50.000000" : 31360,
"60.000000" : 32128,
"70.000000" : 33024,
"80.000000" : 35072,
"90.000000" : 44800,
"95.000000" : 52480,
"99.000000" : 62720,
"99.500000" : 68096,
"99.900000" : 130560,
"99.950000" : 142336,
"99.990000" : 158720
}
},
"lat_ns" : {
"min" : 15499,
"max" : 207278,
"mean" : 35450.237329,
"stddev" : 9103.650064,
"N" : 1568096
},
"bw_min" : 894984,
"bw_max" : 897680,
"bw_agg" : 100.000000,
"bw_mean" : 896217.000000,
"bw_dev" : 865.985877,
"bw_samples" : 14,
"iops_min" : 223746,
"iops_max" : 224420,
"iops_mean" : 224054.214286,
"iops_stddev" : 216.539276,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 19.214286,
"sys_cpu" : 54.914286,
"ctx" : 652730,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.291055,
"50" : 92.258539,
"100" : 7.203864,
"250" : 0.246989,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=90091: Tue Oct 25 23:54:26 2022
read: IOPS=224k, BW=875MiB/s (917MB/s)(6125MiB/7001msec)
slat (nsec): min=1482, max=25568, avg=1774.25, stdev=161.38
clat (usec): min=13, max=201, avg=33.68, stdev= 9.11
lat (usec): min=15, max=207, avg=35.45, stdev= 9.10
clat percentiles (usec):
| 1.00th=[ 24], 5.00th=[ 26], 10.00th=[ 28], 20.00th=[ 30],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 33],
| 70.00th=[ 34], 80.00th=[ 36], 90.00th=[ 45], 95.00th=[ 53],
| 99.00th=[ 63], 99.50th=[ 69], 99.90th=[ 131], 99.95th=[ 143],
| 99.99th=[ 159]
bw ( KiB/s): min=894984, max=897680, per=100.00%, avg=896217.00, stdev=865.99, samples=14
iops : min=223746, max=224420, avg=224054.21, stdev=216.54, samples=14
lat (usec) : 20=0.29%, 50=92.26%, 100=7.20%, 250=0.25%
cpu : usr=19.21%, sys=54.91%, ctx=652730, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1568089,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6125MiB (6423MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_8_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-xnvme_2.txt"
$ cat /tmp/fio-output_4K_8_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666734879,
"timestamp_ms" : 1666734879675,
"time" : "Tue Oct 25 23:54:39 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 6417117184,
"io_kbytes" : 6266716,
"bw_bytes" : 916600083,
"bw" : 895117,
"iops" : 223778.317383,
"runtime" : 7001,
"total_ios" : 1566672,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1452,
"max" : 42480,
"mean" : 2043.000890,
"stddev" : 749.595984,
"N" : 1566672
},
"clat_ns" : {
"min" : 440,
"max" : 228789,
"mean" : 33399.843687,
"stddev" : 8968.879387,
"N" : 1566679,
"percentile" : {
"1.000000" : 23168,
"5.000000" : 27008,
"10.000000" : 27520,
"20.000000" : 28544,
"30.000000" : 29824,
"40.000000" : 31104,
"50.000000" : 31360,
"60.000000" : 31616,
"70.000000" : 32384,
"80.000000" : 34048,
"90.000000" : 44800,
"95.000000" : 51968,
"99.000000" : 62208,
"99.500000" : 67072,
"99.900000" : 132096,
"99.950000" : 140288,
"99.990000" : 156672
}
},
"lat_ns" : {
"min" : 18515,
"max" : 230592,
"mean" : 35442.843629,
"stddev" : 8913.795051,
"N" : 1566679
},
"bw_min" : 893096,
"bw_max" : 897528,
"bw_agg" : 100.000000,
"bw_mean" : 895350.214286,
"bw_dev" : 1168.143970,
"bw_samples" : 14,
"iops_min" : 223274,
"iops_max" : 224382,
"iops_mean" : 223837.500000,
"iops_stddev" : 292.004413,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 23.857143,
"sys_cpu" : 59.200000,
"ctx" : 370799,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999968,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 0.034468,
"50" : 92.821854,
"100" : 6.875274,
"250" : 0.268595,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=90132: Tue Oct 25 23:54:39 2022
read: IOPS=224k, BW=874MiB/s (917MB/s)(6120MiB/7001msec)
slat (nsec): min=1452, max=42480, avg=2043.00, stdev=749.60
clat (nsec): min=440, max=228789, avg=33399.84, stdev=8968.88
lat (usec): min=18, max=230, avg=35.44, stdev= 8.91
clat percentiles (usec):
| 1.00th=[ 24], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 29],
| 30.00th=[ 30], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 33], 80.00th=[ 35], 90.00th=[ 45], 95.00th=[ 52],
| 99.00th=[ 63], 99.50th=[ 68], 99.90th=[ 133], 99.95th=[ 141],
| 99.99th=[ 157]
bw ( KiB/s): min=893096, max=897528, per=100.00%, avg=895350.21, stdev=1168.14, samples=14
iops : min=223274, max=224382, avg=223837.50, stdev=292.00, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=0.03%, 50=92.82%, 100=6.88%
lat (usec) : 250=0.27%
cpu : usr=23.86%, sys=59.20%, ctx=370799, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1566672,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6120MiB (6417MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733187,
"timestamp_ms" : 1666733187146,
"time" : "Tue Oct 25 23:26:27 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2400022528,
"io_kbytes" : 2343772,
"bw_bytes" : 342811388,
"bw" : 334776,
"iops" : 83694.186545,
"runtime" : 7001,
"total_ios" : 585943,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 160,
"max" : 9938,
"mean" : 176.388493,
"stddev" : 26.408079,
"N" : 585943
},
"clat_ns" : {
"min" : 10430,
"max" : 113473,
"mean" : 11527.381930,
"stddev" : 2234.725223,
"N" : 585943,
"percentile" : {
"1.000000" : 10944,
"5.000000" : 11072,
"10.000000" : 11072,
"20.000000" : 11072,
"30.000000" : 11072,
"40.000000" : 11072,
"50.000000" : 11200,
"60.000000" : 11200,
"70.000000" : 11328,
"80.000000" : 11328,
"90.000000" : 11456,
"95.000000" : 12096,
"99.000000" : 22400,
"99.500000" : 28288,
"99.900000" : 28800,
"99.950000" : 44288,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 10610,
"max" : 113653,
"mean" : 11703.770423,
"stddev" : 2235.118972,
"N" : 585943
},
"bw_min" : 334472,
"bw_max" : 336208,
"bw_agg" : 100.000000,
"bw_mean" : 334961.142857,
"bw_dev" : 434.449228,
"bw_samples" : 14,
"iops_min" : 83618,
"iops_max" : 84052,
"iops_mean" : 83740.214286,
"iops_stddev" : 108.614476,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 80.728571,
"sys_cpu" : 19.257143,
"ctx" : 15,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 98.614541,
"50" : 1.340915,
"100" : 0.044373,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=84947: Tue Oct 25 23:26:27 2022
read: IOPS=83.7k, BW=327MiB/s (343MB/s)(2289MiB/7001msec)
slat (nsec): min=160, max=9938, avg=176.39, stdev=26.41
clat (usec): min=10, max=113, avg=11.53, stdev= 2.23
lat (usec): min=10, max=113, avg=11.70, stdev= 2.24
clat percentiles (nsec):
| 1.00th=[10944], 5.00th=[11072], 10.00th=[11072], 20.00th=[11072],
| 30.00th=[11072], 40.00th=[11072], 50.00th=[11200], 60.00th=[11200],
| 70.00th=[11328], 80.00th=[11328], 90.00th=[11456], 95.00th=[12096],
| 99.00th=[22400], 99.50th=[28288], 99.90th=[28800], 99.95th=[44288],
| 99.99th=[68096]
bw ( KiB/s): min=334472, max=336208, per=100.00%, avg=334961.14, stdev=434.45, samples=14
iops : min=83618, max=84052, avg=83740.21, stdev=108.61, samples=14
lat (usec) : 20=98.61%, 50=1.34%, 100=0.04%, 250=0.01%
cpu : usr=80.73%, sys=19.26%, ctx=15, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=585943,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=327MiB/s (343MB/s), 327MiB/s-327MiB/s (343MB/s-343MB/s), io=2289MiB (2400MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733202,
"timestamp_ms" : 1666733202256,
"time" : "Tue Oct 25 23:26:42 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2856644608,
"io_kbytes" : 2789692,
"bw_bytes" : 408033796,
"bw" : 398470,
"iops" : 99617.626053,
"runtime" : 7001,
"total_ios" : 697423,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 161,
"max" : 12935,
"mean" : 181.008962,
"stddev" : 43.708701,
"N" : 697423
},
"clat_ns" : {
"min" : 9037,
"max" : 95539,
"mean" : 9612.561520,
"stddev" : 1943.039396,
"N" : 697423,
"percentile" : {
"1.000000" : 9152,
"5.000000" : 9152,
"10.000000" : 9152,
"20.000000" : 9152,
"30.000000" : 9280,
"40.000000" : 9280,
"50.000000" : 9280,
"60.000000" : 9408,
"70.000000" : 9408,
"80.000000" : 9408,
"90.000000" : 9408,
"95.000000" : 10048,
"99.000000" : 18816,
"99.500000" : 20864,
"99.900000" : 25472,
"99.950000" : 42752,
"99.990000" : 67072
}
},
"lat_ns" : {
"min" : 9218,
"max" : 95719,
"mean" : 9793.570482,
"stddev" : 1943.679350,
"N" : 697423
},
"bw_min" : 397920,
"bw_max" : 399024,
"bw_agg" : 100.000000,
"bw_mean" : 398570.642857,
"bw_dev" : 358.930807,
"bw_samples" : 14,
"iops_min" : 99480,
"iops_max" : 99756,
"iops_mean" : 99642.642857,
"iops_stddev" : 89.712024,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 70.271429,
"sys_cpu" : 29.700000,
"ctx" : 8,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 94.961594,
"20" : 4.229141,
"50" : 0.766393,
"100" : 0.042872,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=84996: Tue Oct 25 23:26:42 2022
read: IOPS=99.6k, BW=389MiB/s (408MB/s)(2724MiB/7001msec)
slat (nsec): min=161, max=12935, avg=181.01, stdev=43.71
clat (nsec): min=9037, max=95539, avg=9612.56, stdev=1943.04
lat (nsec): min=9218, max=95719, avg=9793.57, stdev=1943.68
clat percentiles (nsec):
| 1.00th=[ 9152], 5.00th=[ 9152], 10.00th=[ 9152], 20.00th=[ 9152],
| 30.00th=[ 9280], 40.00th=[ 9280], 50.00th=[ 9280], 60.00th=[ 9408],
| 70.00th=[ 9408], 80.00th=[ 9408], 90.00th=[ 9408], 95.00th=[10048],
| 99.00th=[18816], 99.50th=[20864], 99.90th=[25472], 99.95th=[42752],
| 99.99th=[67072]
bw ( KiB/s): min=397920, max=399024, per=100.00%, avg=398570.64, stdev=358.93, samples=14
iops : min=99480, max=99756, avg=99642.64, stdev=89.71, samples=14
lat (usec) : 10=94.96%, 20=4.23%, 50=0.77%, 100=0.04%
cpu : usr=70.27%, sys=29.70%, ctx=8, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=697423,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=389MiB/s (408MB/s), 389MiB/s-389MiB/s (408MB/s-408MB/s), io=2724MiB (2857MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733217,
"timestamp_ms" : 1666733217255,
"time" : "Tue Oct 25 23:26:57 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2650079232,
"io_kbytes" : 2587968,
"bw_bytes" : 378528671,
"bw" : 369656,
"iops" : 92414.226539,
"runtime" : 7001,
"total_ios" : 646992,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1673,
"max" : 15529,
"mean" : 1800.539421,
"stddev" : 116.222659,
"N" : 646992
},
"clat_ns" : {
"min" : 251,
"max" : 115536,
"mean" : 8778.187362,
"stddev" : 1981.005506,
"N" : 646992,
"percentile" : {
"1.000000" : 8256,
"5.000000" : 8384,
"10.000000" : 8384,
"20.000000" : 8384,
"30.000000" : 8384,
"40.000000" : 8384,
"50.000000" : 8512,
"60.000000" : 8512,
"70.000000" : 8512,
"80.000000" : 8512,
"90.000000" : 8640,
"95.000000" : 9408,
"99.000000" : 18816,
"99.500000" : 20352,
"99.900000" : 24960,
"99.950000" : 41728,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 9598,
"max" : 117290,
"mean" : 10578.726783,
"stddev" : 1983.792726,
"N" : 646992
},
"bw_min" : 367720,
"bw_max" : 370808,
"bw_agg" : 100.000000,
"bw_mean" : 369810.769231,
"bw_dev" : 877.410504,
"bw_samples" : 13,
"iops_min" : 91930,
"iops_max" : 92702,
"iops_mean" : 92452.615385,
"iops_stddev" : 219.323406,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 5.642857,
"sys_cpu" : 94.328571,
"ctx" : 14,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 95.804121,
"20" : 3.551049,
"50" : 0.603562,
"100" : 0.036940,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=85046: Tue Oct 25 23:26:57 2022
read: IOPS=92.4k, BW=361MiB/s (379MB/s)(2527MiB/7001msec)
slat (nsec): min=1673, max=15529, avg=1800.54, stdev=116.22
clat (nsec): min=251, max=115536, avg=8778.19, stdev=1981.01
lat (usec): min=9, max=117, avg=10.58, stdev= 1.98
clat percentiles (nsec):
| 1.00th=[ 8256], 5.00th=[ 8384], 10.00th=[ 8384], 20.00th=[ 8384],
| 30.00th=[ 8384], 40.00th=[ 8384], 50.00th=[ 8512], 60.00th=[ 8512],
| 70.00th=[ 8512], 80.00th=[ 8512], 90.00th=[ 8640], 95.00th=[ 9408],
| 99.00th=[18816], 99.50th=[20352], 99.90th=[24960], 99.95th=[41728],
| 99.99th=[66048]
bw ( KiB/s): min=367720, max=370808, per=100.00%, avg=369810.77, stdev=877.41, samples=13
iops : min=91930, max=92702, avg=92452.62, stdev=219.32, samples=13
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 4=0.01%, 10=95.80%, 20=3.55%, 50=0.60%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=5.64%, sys=94.33%, ctx=14, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=646992,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=361MiB/s (379MB/s), 361MiB/s-361MiB/s (379MB/s-379MB/s), io=2527MiB (2650MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733232,
"timestamp_ms" : 1666733232288,
"time" : "Tue Oct 25 23:27:12 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2934267904,
"io_kbytes" : 2865496,
"bw_bytes" : 419121254,
"bw" : 409298,
"iops" : 102324.525068,
"runtime" : 7001,
"total_ios" : 716374,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1543,
"max" : 18425,
"mean" : 1610.136684,
"stddev" : 115.405056,
"N" : 716374
},
"clat_ns" : {
"min" : 400,
"max" : 1295729,
"mean" : 7923.244064,
"stddev" : 2451.556179,
"N" : 716374,
"percentile" : {
"1.000000" : 7392,
"5.000000" : 7456,
"10.000000" : 7456,
"20.000000" : 7520,
"30.000000" : 7520,
"40.000000" : 7584,
"50.000000" : 7648,
"60.000000" : 7648,
"70.000000" : 7712,
"80.000000" : 7712,
"90.000000" : 7776,
"95.000000" : 8160,
"99.000000" : 17280,
"99.500000" : 20096,
"99.900000" : 22656,
"99.950000" : 32384,
"99.990000" : 65280
}
},
"lat_ns" : {
"min" : 8926,
"max" : 1297322,
"mean" : 9533.380748,
"stddev" : 2454.613923,
"N" : 716374
},
"bw_min" : 408576,
"bw_max" : 410869,
"bw_agg" : 100.000000,
"bw_mean" : 409391.785714,
"bw_dev" : 583.681180,
"bw_samples" : 14,
"iops_min" : 102144,
"iops_max" : 102717,
"iops_mean" : 102347.928571,
"iops_stddev" : 145.871632,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 6.957143,
"sys_cpu" : 93.000000,
"ctx" : 48,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.128419,
"20" : 3.324520,
"50" : 0.511325,
"100" : 0.034619,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=85097: Tue Oct 25 23:27:12 2022
read: IOPS=102k, BW=400MiB/s (419MB/s)(2798MiB/7001msec)
slat (nsec): min=1543, max=18425, avg=1610.14, stdev=115.41
clat (nsec): min=400, max=1295.7k, avg=7923.24, stdev=2451.56
lat (usec): min=8, max=1297, avg= 9.53, stdev= 2.45
clat percentiles (nsec):
| 1.00th=[ 7392], 5.00th=[ 7456], 10.00th=[ 7456], 20.00th=[ 7520],
| 30.00th=[ 7520], 40.00th=[ 7584], 50.00th=[ 7648], 60.00th=[ 7648],
| 70.00th=[ 7712], 80.00th=[ 7712], 90.00th=[ 7776], 95.00th=[ 8160],
| 99.00th=[17280], 99.50th=[20096], 99.90th=[22656], 99.95th=[32384],
| 99.99th=[65280]
bw ( KiB/s): min=408576, max=410869, per=100.00%, avg=409391.79, stdev=583.68, samples=14
iops : min=102144, max=102717, avg=102347.93, stdev=145.87, samples=14
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 4=0.01%, 10=96.13%, 20=3.32%, 50=0.51%, 100=0.03%
lat (usec) : 250=0.01%
lat (msec) : 2=0.01%
cpu : usr=6.96%, sys=93.00%, ctx=48, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=716374,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=400MiB/s (419MB/s), 400MiB/s-400MiB/s (419MB/s-419MB/s), io=2798MiB (2934MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733247,
"timestamp_ms" : 1666733247334,
"time" : "Tue Oct 25 23:27:27 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2939764736,
"io_kbytes" : 2870864,
"bw_bytes" : 419906404,
"bw" : 410064,
"iops" : 102516.211970,
"runtime" : 7001,
"total_ios" : 717716,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1522,
"max" : 14347,
"mean" : 1578.864615,
"stddev" : 120.389006,
"N" : 717716
},
"clat_ns" : {
"min" : 241,
"max" : 96561,
"mean" : 7938.478975,
"stddev" : 1946.448668,
"N" : 717716,
"percentile" : {
"1.000000" : 7456,
"5.000000" : 7456,
"10.000000" : 7456,
"20.000000" : 7520,
"30.000000" : 7520,
"40.000000" : 7584,
"50.000000" : 7648,
"60.000000" : 7648,
"70.000000" : 7712,
"80.000000" : 7712,
"90.000000" : 7776,
"95.000000" : 8160,
"99.000000" : 17280,
"99.500000" : 20352,
"99.900000" : 22912,
"99.950000" : 32640,
"99.990000" : 65280
}
},
"lat_ns" : {
"min" : 8926,
"max" : 98124,
"mean" : 9517.343590,
"stddev" : 1950.790720,
"N" : 717716
},
"bw_min" : 409424,
"bw_max" : 411160,
"bw_agg" : 100.000000,
"bw_mean" : 410161.928571,
"bw_dev" : 477.509318,
"bw_samples" : 14,
"iops_min" : 102354,
"iops_max" : 102790,
"iops_mean" : 102540.428571,
"iops_stddev" : 119.741204,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 8.157143,
"sys_cpu" : 91.814286,
"ctx" : 5,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.122422,
"20" : 3.278456,
"50" : 0.559135,
"100" : 0.038316,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=85148: Tue Oct 25 23:27:27 2022
read: IOPS=103k, BW=400MiB/s (420MB/s)(2804MiB/7001msec)
slat (nsec): min=1522, max=14347, avg=1578.86, stdev=120.39
clat (nsec): min=241, max=96561, avg=7938.48, stdev=1946.45
lat (nsec): min=8926, max=98124, avg=9517.34, stdev=1950.79
clat percentiles (nsec):
| 1.00th=[ 7456], 5.00th=[ 7456], 10.00th=[ 7456], 20.00th=[ 7520],
| 30.00th=[ 7520], 40.00th=[ 7584], 50.00th=[ 7648], 60.00th=[ 7648],
| 70.00th=[ 7712], 80.00th=[ 7712], 90.00th=[ 7776], 95.00th=[ 8160],
| 99.00th=[17280], 99.50th=[20352], 99.90th=[22912], 99.95th=[32640],
| 99.99th=[65280]
bw ( KiB/s): min=409424, max=411160, per=100.00%, avg=410161.93, stdev=477.51, samples=14
iops : min=102354, max=102790, avg=102540.43, stdev=119.74, samples=14
lat (nsec) : 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 4=0.01%, 10=96.12%, 20=3.28%, 50=0.56%, 100=0.04%
cpu : usr=8.16%, sys=91.81%, ctx=5, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=717716,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=400MiB/s (420MB/s), 400MiB/s-400MiB/s (420MB/s-420MB/s), io=2804MiB (2940MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-xnvme_0.txt"
$ cat /tmp/fio-output_4K_1_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733261,
"timestamp_ms" : 1666733261012,
"time" : "Tue Oct 25 23:27:41 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 1795207168,
"io_kbytes" : 1753132,
"bw_bytes" : 256421535,
"bw" : 250411,
"iops" : 62602.913869,
"runtime" : 7001,
"total_ios" : 438283,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2655,
"max" : 27832,
"mean" : 2842.061771,
"stddev" : 230.543540,
"N" : 438283
},
"clat_ns" : {
"min" : 281,
"max" : 122980,
"mean" : 12554.664060,
"stddev" : 2139.295770,
"N" : 438283,
"percentile" : {
"1.000000" : 11968,
"5.000000" : 12096,
"10.000000" : 12096,
"20.000000" : 12096,
"30.000000" : 12224,
"40.000000" : 12224,
"50.000000" : 12224,
"60.000000" : 12352,
"70.000000" : 12352,
"80.000000" : 12352,
"90.000000" : 12480,
"95.000000" : 12608,
"99.000000" : 22144,
"99.500000" : 28288,
"99.900000" : 40704,
"99.950000" : 54016,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 12182,
"max" : 125816,
"mean" : 15396.725830,
"stddev" : 2157.279613,
"N" : 438283
},
"bw_min" : 249896,
"bw_max" : 251094,
"bw_agg" : 100.000000,
"bw_mean" : 250470.142857,
"bw_dev" : 335.320250,
"bw_samples" : 14,
"iops_min" : 62472,
"iops_max" : 62773,
"iops_mean" : 62617.500000,
"iops_stddev" : 83.901728,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 10.585714,
"sys_cpu" : 38.442857,
"ctx" : 438270,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 98.596113,
"50" : 1.331788,
"100" : 0.064798,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=85187: Tue Oct 25 23:27:41 2022
read: IOPS=62.6k, BW=245MiB/s (256MB/s)(1712MiB/7001msec)
slat (nsec): min=2655, max=27832, avg=2842.06, stdev=230.54
clat (nsec): min=281, max=122980, avg=12554.66, stdev=2139.30
lat (usec): min=12, max=125, avg=15.40, stdev= 2.16
clat percentiles (nsec):
| 1.00th=[11968], 5.00th=[12096], 10.00th=[12096], 20.00th=[12096],
| 30.00th=[12224], 40.00th=[12224], 50.00th=[12224], 60.00th=[12352],
| 70.00th=[12352], 80.00th=[12352], 90.00th=[12480], 95.00th=[12608],
| 99.00th=[22144], 99.50th=[28288], 99.90th=[40704], 99.95th=[54016],
| 99.99th=[68096]
bw ( KiB/s): min=249896, max=251094, per=100.00%, avg=250470.14, stdev=335.32, samples=14
iops : min=62472, max=62773, avg=62617.50, stdev=83.90, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=98.60%, 50=1.33%, 100=0.06%
lat (usec) : 250=0.01%
cpu : usr=10.59%, sys=38.44%, ctx=438270, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=438283,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=245MiB/s (256MB/s), 245MiB/s-245MiB/s (256MB/s-256MB/s), io=1712MiB (1795MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-xnvme_1.txt"
$ cat /tmp/fio-output_4K_1_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733274,
"timestamp_ms" : 1666733274648,
"time" : "Tue Oct 25 23:27:54 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 1804898304,
"io_kbytes" : 1762596,
"bw_bytes" : 257805785,
"bw" : 251763,
"iops" : 62940.865591,
"runtime" : 7001,
"total_ios" : 440649,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2645,
"max" : 27261,
"mean" : 2816.685162,
"stddev" : 258.194088,
"N" : 440649
},
"clat_ns" : {
"min" : 380,
"max" : 467586,
"mean" : 12522.016850,
"stddev" : 2175.039916,
"N" : 440649,
"percentile" : {
"1.000000" : 11968,
"5.000000" : 12096,
"10.000000" : 12096,
"20.000000" : 12096,
"30.000000" : 12224,
"40.000000" : 12224,
"50.000000" : 12224,
"60.000000" : 12352,
"70.000000" : 12352,
"80.000000" : 12352,
"90.000000" : 12480,
"95.000000" : 12608,
"99.000000" : 22144,
"99.500000" : 28032,
"99.900000" : 33536,
"99.950000" : 50432,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 12243,
"max" : 470382,
"mean" : 15338.702012,
"stddev" : 2192.213103,
"N" : 440649
},
"bw_min" : 250829,
"bw_max" : 252600,
"bw_agg" : 100.000000,
"bw_mean" : 251826.071429,
"bw_dev" : 417.064921,
"bw_samples" : 14,
"iops_min" : 62707,
"iops_max" : 63150,
"iops_mean" : 62956.500000,
"iops_stddev" : 104.312216,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 11.042857,
"sys_cpu" : 38.328571,
"ctx" : 440630,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 98.638372,
"50" : 1.295589,
"100" : 0.056054,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=85227: Tue Oct 25 23:27:54 2022
read: IOPS=62.9k, BW=246MiB/s (258MB/s)(1721MiB/7001msec)
slat (nsec): min=2645, max=27261, avg=2816.69, stdev=258.19
clat (nsec): min=380, max=467586, avg=12522.02, stdev=2175.04
lat (usec): min=12, max=470, avg=15.34, stdev= 2.19
clat percentiles (nsec):
| 1.00th=[11968], 5.00th=[12096], 10.00th=[12096], 20.00th=[12096],
| 30.00th=[12224], 40.00th=[12224], 50.00th=[12224], 60.00th=[12352],
| 70.00th=[12352], 80.00th=[12352], 90.00th=[12480], 95.00th=[12608],
| 99.00th=[22144], 99.50th=[28032], 99.90th=[33536], 99.95th=[50432],
| 99.99th=[66048]
bw ( KiB/s): min=250829, max=252600, per=100.00%, avg=251826.07, stdev=417.06, samples=14
iops : min=62707, max=63150, avg=62956.50, stdev=104.31, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=98.64%, 50=1.30%, 100=0.06%
lat (usec) : 250=0.01%, 500=0.01%
cpu : usr=11.04%, sys=38.33%, ctx=440630, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=440649,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=246MiB/s (258MB/s), 246MiB/s-246MiB/s (258MB/s-258MB/s), io=1721MiB (1805MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-xnvme_2.txt"
$ cat /tmp/fio-output_4K_1_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733288,
"timestamp_ms" : 1666733288284,
"time" : "Tue Oct 25 23:28:08 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 1813983232,
"io_kbytes" : 1771468,
"bw_bytes" : 259103446,
"bw" : 253030,
"iops" : 63257.534638,
"runtime" : 7001,
"total_ios" : 442866,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2584,
"max" : 28093,
"mean" : 2753.253893,
"stddev" : 286.498944,
"N" : 442867
},
"clat_ns" : {
"min" : 421,
"max" : 181340,
"mean" : 12481.790944,
"stddev" : 2028.063473,
"N" : 442866,
"percentile" : {
"1.000000" : 11968,
"5.000000" : 11968,
"10.000000" : 11968,
"20.000000" : 12096,
"30.000000" : 12096,
"40.000000" : 12224,
"50.000000" : 12224,
"60.000000" : 12224,
"70.000000" : 12352,
"80.000000" : 12352,
"90.000000" : 12352,
"95.000000" : 12608,
"99.000000" : 22144,
"99.500000" : 27776,
"99.900000" : 32384,
"99.950000" : 50432,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 12102,
"max" : 184025,
"mean" : 15235.045014,
"stddev" : 2052.149286,
"N" : 442866
},
"bw_min" : 252120,
"bw_max" : 254468,
"bw_agg" : 100.000000,
"bw_mean" : 253092.857143,
"bw_dev" : 627.819040,
"bw_samples" : 14,
"iops_min" : 63030,
"iops_max" : 63617,
"iops_mean" : 63273.214286,
"iops_stddev" : 156.954760,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 10.757143,
"sys_cpu" : 38.157143,
"ctx" : 442837,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 98.632995,
"50" : 1.304909,
"100" : 0.052612,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=85266: Tue Oct 25 23:28:08 2022
read: IOPS=63.3k, BW=247MiB/s (259MB/s)(1730MiB/7001msec)
slat (nsec): min=2584, max=28093, avg=2753.25, stdev=286.50
clat (nsec): min=421, max=181340, avg=12481.79, stdev=2028.06
lat (usec): min=12, max=184, avg=15.24, stdev= 2.05
clat percentiles (nsec):
| 1.00th=[11968], 5.00th=[11968], 10.00th=[11968], 20.00th=[12096],
| 30.00th=[12096], 40.00th=[12224], 50.00th=[12224], 60.00th=[12224],
| 70.00th=[12352], 80.00th=[12352], 90.00th=[12352], 95.00th=[12608],
| 99.00th=[22144], 99.50th=[27776], 99.90th=[32384], 99.95th=[50432],
| 99.99th=[66048]
bw ( KiB/s): min=252120, max=254468, per=100.00%, avg=253092.86, stdev=627.82, samples=14
iops : min=63030, max=63617, avg=63273.21, stdev=156.95, samples=14
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 10=0.01%, 20=98.63%, 50=1.30%, 100=0.05%, 250=0.01%
cpu : usr=10.76%, sys=38.16%, ctx=442837, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=442866,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=247MiB/s (259MB/s), 247MiB/s-247MiB/s (259MB/s-259MB/s), io=1730MiB (1814MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-reference_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-reference_0.txt"
$ cat /tmp/fio-output_4K_1_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733301,
"timestamp_ms" : 1666733301938,
"time" : "Tue Oct 25 23:28:21 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 1673572352,
"io_kbytes" : 1634348,
"bw_bytes" : 239047614,
"bw" : 233444,
"iops" : 58361.234109,
"runtime" : 7001,
"total_ios" : 408587,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2836,
"max" : 17292,
"mean" : 3074.658611,
"stddev" : 158.340253,
"N" : 408587
},
"clat_ns" : {
"min" : 802,
"max" : 118943,
"mean" : 13512.992719,
"stddev" : 2095.582789,
"N" : 408587,
"percentile" : {
"1.000000" : 12864,
"5.000000" : 12992,
"10.000000" : 13120,
"20.000000" : 13120,
"30.000000" : 13248,
"40.000000" : 13248,
"50.000000" : 13248,
"60.000000" : 13248,
"70.000000" : 13248,
"80.000000" : 13376,
"90.000000" : 13376,
"95.000000" : 13504,
"99.000000" : 23680,
"99.500000" : 28800,
"99.900000" : 33536,
"99.950000" : 50432,
"99.990000" : 71168
}
},
"lat_ns" : {
"min" : 14397,
"max" : 122139,
"mean" : 16587.651330,
"stddev" : 2102.634447,
"N" : 408587
},
"bw_min" : 233168,
"bw_max" : 233976,
"bw_agg" : 100.000000,
"bw_mean" : 233596.714286,
"bw_dev" : 275.558574,
"bw_samples" : 14,
"iops_min" : 58292,
"iops_max" : 58494,
"iops_mean" : 58399.000000,
"iops_stddev" : 68.835367,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 9.100000,
"sys_cpu" : 32.528571,
"ctx" : 408588,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 98.564810,
"50" : 1.376696,
"100" : 0.051641,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 586804,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 5712,
"write_ticks" : 0,
"in_queue" : 5712,
"util" : 99.108381
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=85307: Tue Oct 25 23:28:21 2022
read: IOPS=58.4k, BW=228MiB/s (239MB/s)(1596MiB/7001msec)
slat (nsec): min=2836, max=17292, avg=3074.66, stdev=158.34
clat (nsec): min=802, max=118943, avg=13512.99, stdev=2095.58
lat (usec): min=14, max=122, avg=16.59, stdev= 2.10
clat percentiles (nsec):
| 1.00th=[12864], 5.00th=[12992], 10.00th=[13120], 20.00th=[13120],
| 30.00th=[13248], 40.00th=[13248], 50.00th=[13248], 60.00th=[13248],
| 70.00th=[13248], 80.00th=[13376], 90.00th=[13376], 95.00th=[13504],
| 99.00th=[23680], 99.50th=[28800], 99.90th=[33536], 99.95th=[50432],
| 99.99th=[71168]
bw ( KiB/s): min=233168, max=233976, per=100.00%, avg=233596.71, stdev=275.56, samples=14
iops : min=58292, max=58494, avg=58399.00, stdev=68.84, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 10=0.01%, 20=98.56%, 50=1.38%, 100=0.05%, 250=0.01%
cpu : usr=9.10%, sys=32.53%, ctx=408588, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=408587,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=228MiB/s (239MB/s), 228MiB/s-228MiB/s (239MB/s-239MB/s), io=1596MiB (1674MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=586804/0, merge=0/0, ticks=5712/0, in_queue=5712, util=99.11%
$ rm /tmp/fio-output_4K_1_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-reference_1.txt"
$ cat /tmp/fio-output_4K_1_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733315,
"timestamp_ms" : 1666733315587,
"time" : "Tue Oct 25 23:28:35 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 1801781248,
"io_kbytes" : 1759552,
"bw_bytes" : 257360555,
"bw" : 251328,
"iops" : 62832.166833,
"runtime" : 7001,
"total_ios" : 439888,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2765,
"max" : 26921,
"mean" : 2932.077167,
"stddev" : 192.262685,
"N" : 439888
},
"clat_ns" : {
"min" : 881,
"max" : 77566,
"mean" : 12358.918272,
"stddev" : 1970.343765,
"N" : 439888,
"percentile" : {
"1.000000" : 11840,
"5.000000" : 11840,
"10.000000" : 11968,
"20.000000" : 11968,
"30.000000" : 11968,
"40.000000" : 12096,
"50.000000" : 12096,
"60.000000" : 12096,
"70.000000" : 12224,
"80.000000" : 12224,
"90.000000" : 12352,
"95.000000" : 12480,
"99.000000" : 21888,
"99.500000" : 27264,
"99.900000" : 31616,
"99.950000" : 49920,
"99.990000" : 67072
}
},
"lat_ns" : {
"min" : 12002,
"max" : 80582,
"mean" : 15290.995440,
"stddev" : 1982.252033,
"N" : 439888
},
"bw_min" : 250264,
"bw_max" : 252440,
"bw_agg" : 100.000000,
"bw_mean" : 251431.928571,
"bw_dev" : 631.720540,
"bw_samples" : 14,
"iops_min" : 62566,
"iops_max" : 63110,
"iops_mean" : 62857.928571,
"iops_stddev" : 157.898928,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 11.057143,
"sys_cpu" : 37.400000,
"ctx" : 439880,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 98.638062,
"50" : 1.307151,
"100" : 0.047057,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 631256,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 5943,
"write_ticks" : 0,
"in_queue" : 5943,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=85345: Tue Oct 25 23:28:35 2022
read: IOPS=62.8k, BW=245MiB/s (257MB/s)(1718MiB/7001msec)
slat (nsec): min=2765, max=26921, avg=2932.08, stdev=192.26
clat (nsec): min=881, max=77566, avg=12358.92, stdev=1970.34
lat (nsec): min=12002, max=80582, avg=15291.00, stdev=1982.25
clat percentiles (nsec):
| 1.00th=[11840], 5.00th=[11840], 10.00th=[11968], 20.00th=[11968],
| 30.00th=[11968], 40.00th=[12096], 50.00th=[12096], 60.00th=[12096],
| 70.00th=[12224], 80.00th=[12224], 90.00th=[12352], 95.00th=[12480],
| 99.00th=[21888], 99.50th=[27264], 99.90th=[31616], 99.95th=[49920],
| 99.99th=[67072]
bw ( KiB/s): min=250264, max=252440, per=100.00%, avg=251431.93, stdev=631.72, samples=14
iops : min=62566, max=63110, avg=62857.93, stdev=157.90, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=98.64%, 50=1.31%, 100=0.05%
cpu : usr=11.06%, sys=37.40%, ctx=439880, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=439888,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=245MiB/s (257MB/s), 245MiB/s-245MiB/s (257MB/s-257MB/s), io=1718MiB (1802MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=631256/0, merge=0/0, ticks=5943/0, in_queue=5943, util=99.09%
$ rm /tmp/fio-output_4K_1_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-reference_2.txt"
$ cat /tmp/fio-output_4K_1_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733329,
"timestamp_ms" : 1666733329246,
"time" : "Tue Oct 25 23:28:49 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 1831936000,
"io_kbytes" : 1789000,
"bw_bytes" : 261667761,
"bw" : 255534,
"iops" : 63883.588059,
"runtime" : 7001,
"total_ios" : 447249,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2605,
"max" : 37320,
"mean" : 2851.803557,
"stddev" : 243.670282,
"N" : 447250
},
"clat_ns" : {
"min" : 872,
"max" : 136185,
"mean" : 12256.182628,
"stddev" : 1988.047396,
"N" : 447249,
"percentile" : {
"1.000000" : 11712,
"5.000000" : 11712,
"10.000000" : 11840,
"20.000000" : 11840,
"30.000000" : 11840,
"40.000000" : 11968,
"50.000000" : 11968,
"60.000000" : 12096,
"70.000000" : 12096,
"80.000000" : 12096,
"90.000000" : 12224,
"95.000000" : 12352,
"99.000000" : 22144,
"99.500000" : 26752,
"99.900000" : 31616,
"99.950000" : 49408,
"99.990000" : 63744
}
},
"lat_ns" : {
"min" : 12163,
"max" : 138971,
"mean" : 15107.986178,
"stddev" : 2008.944842,
"N" : 447249
},
"bw_min" : 254784,
"bw_max" : 257032,
"bw_agg" : 100.000000,
"bw_mean" : 255632.357143,
"bw_dev" : 623.778265,
"bw_samples" : 14,
"iops_min" : 63696,
"iops_max" : 64258,
"iops_mean" : 63908.000000,
"iops_stddev" : 155.974850,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 10.242857,
"sys_cpu" : 37.271429,
"ctx" : 447233,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 98.553155,
"50" : 1.395867,
"100" : 0.044047,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 640196,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 6020,
"write_ticks" : 0,
"in_queue" : 6020,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=85383: Tue Oct 25 23:28:49 2022
read: IOPS=63.9k, BW=250MiB/s (262MB/s)(1747MiB/7001msec)
slat (nsec): min=2605, max=37320, avg=2851.80, stdev=243.67
clat (nsec): min=872, max=136185, avg=12256.18, stdev=1988.05
lat (usec): min=12, max=138, avg=15.11, stdev= 2.01
clat percentiles (nsec):
| 1.00th=[11712], 5.00th=[11712], 10.00th=[11840], 20.00th=[11840],
| 30.00th=[11840], 40.00th=[11968], 50.00th=[11968], 60.00th=[12096],
| 70.00th=[12096], 80.00th=[12096], 90.00th=[12224], 95.00th=[12352],
| 99.00th=[22144], 99.50th=[26752], 99.90th=[31616], 99.95th=[49408],
| 99.99th=[63744]
bw ( KiB/s): min=254784, max=257032, per=100.00%, avg=255632.36, stdev=623.78, samples=14
iops : min=63696, max=64258, avg=63908.00, stdev=155.97, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=98.55%, 50=1.40%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=10.24%, sys=37.27%, ctx=447233, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=447249,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=250MiB/s (262MB/s), 250MiB/s-250MiB/s (262MB/s-262MB/s), io=1747MiB (1832MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=640196/0, merge=0/0, ticks=6020/0, in_queue=6020, util=99.09%
$ rm /tmp/fio-output_4K_1_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-bdev_aio_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-bdev_aio_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733344,
"timestamp_ms" : 1666733344307,
"time" : "Tue Oct 25 23:29:04 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2908217344,
"io_kbytes" : 2840056,
"bw_bytes" : 415400277,
"bw" : 405664,
"iops" : 101416.083417,
"runtime" : 7001,
"total_ios" : 710014,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1562,
"max" : 34896,
"mean" : 1634.305671,
"stddev" : 120.389758,
"N" : 710014
},
"clat_ns" : {
"min" : 221,
"max" : 113843,
"mean" : 7995.203314,
"stddev" : 1932.985662,
"N" : 710014,
"percentile" : {
"1.000000" : 7520,
"5.000000" : 7520,
"10.000000" : 7520,
"20.000000" : 7584,
"30.000000" : 7584,
"40.000000" : 7648,
"50.000000" : 7712,
"60.000000" : 7712,
"70.000000" : 7776,
"80.000000" : 7840,
"90.000000" : 7840,
"95.000000" : 8384,
"99.000000" : 17280,
"99.500000" : 19840,
"99.900000" : 23680,
"99.950000" : 40704,
"99.990000" : 65280
}
},
"lat_ns" : {
"min" : 9027,
"max" : 115476,
"mean" : 9629.508986,
"stddev" : 1936.932387,
"N" : 710014
},
"bw_min" : 404904,
"bw_max" : 407032,
"bw_agg" : 100.000000,
"bw_mean" : 405763.785714,
"bw_dev" : 726.004893,
"bw_samples" : 14,
"iops_min" : 101226,
"iops_max" : 101758,
"iops_mean" : 101440.928571,
"iops_stddev" : 181.477044,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 85.300000,
"sys_cpu" : 14.671429,
"ctx" : 18,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.102471,
"20" : 3.421904,
"50" : 0.436611,
"100" : 0.037605,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=85432: Tue Oct 25 23:29:04 2022
read: IOPS=101k, BW=396MiB/s (415MB/s)(2773MiB/7001msec)
slat (nsec): min=1562, max=34896, avg=1634.31, stdev=120.39
clat (nsec): min=221, max=113843, avg=7995.20, stdev=1932.99
lat (usec): min=9, max=115, avg= 9.63, stdev= 1.94
clat percentiles (nsec):
| 1.00th=[ 7520], 5.00th=[ 7520], 10.00th=[ 7520], 20.00th=[ 7584],
| 30.00th=[ 7584], 40.00th=[ 7648], 50.00th=[ 7712], 60.00th=[ 7712],
| 70.00th=[ 7776], 80.00th=[ 7840], 90.00th=[ 7840], 95.00th=[ 8384],
| 99.00th=[17280], 99.50th=[19840], 99.90th=[23680], 99.95th=[40704],
| 99.99th=[65280]
bw ( KiB/s): min=404904, max=407032, per=100.00%, avg=405763.79, stdev=726.00, samples=14
iops : min=101226, max=101758, avg=101440.93, stdev=181.48, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 10=96.10%, 20=3.42%, 50=0.44%, 100=0.04%, 250=0.01%
cpu : usr=85.30%, sys=14.67%, ctx=18, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=710014,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=396MiB/s (415MB/s), 396MiB/s-396MiB/s (415MB/s-415MB/s), io=2773MiB (2908MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-bdev_aio_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-bdev_aio_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733359,
"timestamp_ms" : 1666733359618,
"time" : "Tue Oct 25 23:29:19 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2864140288,
"io_kbytes" : 2797012,
"bw_bytes" : 409104454,
"bw" : 399516,
"iops" : 99879.017283,
"runtime" : 7001,
"total_ios" : 699253,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1643,
"max" : 27060,
"mean" : 1697.425543,
"stddev" : 169.441086,
"N" : 699253
},
"clat_ns" : {
"min" : 210,
"max" : 77946,
"mean" : 8068.355484,
"stddev" : 1872.173034,
"N" : 699253,
"percentile" : {
"1.000000" : 7648,
"5.000000" : 7712,
"10.000000" : 7712,
"20.000000" : 7712,
"30.000000" : 7712,
"40.000000" : 7712,
"50.000000" : 7776,
"60.000000" : 7776,
"70.000000" : 7776,
"80.000000" : 7776,
"90.000000" : 7904,
"95.000000" : 8512,
"99.000000" : 17280,
"99.500000" : 19328,
"99.900000" : 23936,
"99.950000" : 36608,
"99.990000" : 64768
}
},
"lat_ns" : {
"min" : 9247,
"max" : 79629,
"mean" : 9765.781026,
"stddev" : 1879.123838,
"N" : 699253
},
"bw_min" : 398744,
"bw_max" : 400280,
"bw_agg" : 100.000000,
"bw_mean" : 399610.785714,
"bw_dev" : 354.371856,
"bw_samples" : 14,
"iops_min" : 99686,
"iops_max" : 100070,
"iops_mean" : 99902.642857,
"iops_stddev" : 88.578177,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 85.242857,
"sys_cpu" : 14.728571,
"ctx" : 12,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.071522,
"20" : 3.639741,
"50" : 0.253127,
"100" : 0.032034,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=85481: Tue Oct 25 23:29:19 2022
read: IOPS=99.9k, BW=390MiB/s (409MB/s)(2731MiB/7001msec)
slat (nsec): min=1643, max=27060, avg=1697.43, stdev=169.44
clat (nsec): min=210, max=77946, avg=8068.36, stdev=1872.17
lat (nsec): min=9247, max=79629, avg=9765.78, stdev=1879.12
clat percentiles (nsec):
| 1.00th=[ 7648], 5.00th=[ 7712], 10.00th=[ 7712], 20.00th=[ 7712],
| 30.00th=[ 7712], 40.00th=[ 7712], 50.00th=[ 7776], 60.00th=[ 7776],
| 70.00th=[ 7776], 80.00th=[ 7776], 90.00th=[ 7904], 95.00th=[ 8512],
| 99.00th=[17280], 99.50th=[19328], 99.90th=[23936], 99.95th=[36608],
| 99.99th=[64768]
bw ( KiB/s): min=398744, max=400280, per=100.00%, avg=399610.79, stdev=354.37, samples=14
iops : min=99686, max=100070, avg=99902.64, stdev=88.58, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 10=96.07%, 20=3.64%, 50=0.25%, 100=0.03%
cpu : usr=85.24%, sys=14.73%, ctx=12, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=699253,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=390MiB/s (409MB/s), 390MiB/s-390MiB/s (409MB/s-409MB/s), io=2731MiB (2864MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-bdev_aio_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-bdev_aio_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733374,
"timestamp_ms" : 1666733374688,
"time" : "Tue Oct 25 23:29:34 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2917703680,
"io_kbytes" : 2849320,
"bw_bytes" : 416814811,
"bw" : 407045,
"iops" : 101761.428571,
"runtime" : 7000,
"total_ios" : 712330,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1533,
"max" : 13756,
"mean" : 1593.976842,
"stddev" : 115.555755,
"N" : 712330
},
"clat_ns" : {
"min" : 221,
"max" : 75782,
"mean" : 7996.381760,
"stddev" : 1896.986393,
"N" : 712330,
"percentile" : {
"1.000000" : 7520,
"5.000000" : 7520,
"10.000000" : 7520,
"20.000000" : 7584,
"30.000000" : 7584,
"40.000000" : 7648,
"50.000000" : 7712,
"60.000000" : 7712,
"70.000000" : 7776,
"80.000000" : 7840,
"90.000000" : 7840,
"95.000000" : 8256,
"99.000000" : 17280,
"99.500000" : 20096,
"99.900000" : 22656,
"99.950000" : 32384,
"99.990000" : 65280
}
},
"lat_ns" : {
"min" : 9007,
"max" : 77365,
"mean" : 9590.358602,
"stddev" : 1900.929895,
"N" : 712330
},
"bw_min" : 406344,
"bw_max" : 407800,
"bw_agg" : 100.000000,
"bw_mean" : 407090.071429,
"bw_dev" : 420.417094,
"bw_samples" : 14,
"iops_min" : 101586,
"iops_max" : 101950,
"iops_mean" : 101772.500000,
"iops_stddev" : 105.118797,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 85.871429,
"sys_cpu" : 14.100000,
"ctx" : 10,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.124268,
"20" : 3.367821,
"50" : 0.472955,
"100" : 0.033692,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=85531: Tue Oct 25 23:29:34 2022
read: IOPS=102k, BW=398MiB/s (417MB/s)(2783MiB/7000msec)
slat (nsec): min=1533, max=13756, avg=1593.98, stdev=115.56
clat (nsec): min=221, max=75782, avg=7996.38, stdev=1896.99
lat (nsec): min=9007, max=77365, avg=9590.36, stdev=1900.93
clat percentiles (nsec):
| 1.00th=[ 7520], 5.00th=[ 7520], 10.00th=[ 7520], 20.00th=[ 7584],
| 30.00th=[ 7584], 40.00th=[ 7648], 50.00th=[ 7712], 60.00th=[ 7712],
| 70.00th=[ 7776], 80.00th=[ 7840], 90.00th=[ 7840], 95.00th=[ 8256],
| 99.00th=[17280], 99.50th=[20096], 99.90th=[22656], 99.95th=[32384],
| 99.99th=[65280]
bw ( KiB/s): min=406344, max=407800, per=100.00%, avg=407090.07, stdev=420.42, samples=14
iops : min=101586, max=101950, avg=101772.50, stdev=105.12, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=96.12%, 20=3.37%, 50=0.47%, 100=0.03%
cpu : usr=85.87%, sys=14.10%, ctx=10, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=712330,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=398MiB/s (417MB/s), 398MiB/s-398MiB/s (417MB/s-417MB/s), io=2783MiB (2918MB), run=7000-7000msec
$ rm /tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733389,
"timestamp_ms" : 1666733389939,
"time" : "Tue Oct 25 23:29:49 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2542465024,
"io_kbytes" : 2482876,
"bw_bytes" : 363157409,
"bw" : 354645,
"iops" : 88661.476932,
"runtime" : 7001,
"total_ios" : 620719,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1783,
"max" : 14928,
"mean" : 1926.784324,
"stddev" : 147.911807,
"N" : 620719
},
"clat_ns" : {
"min" : 611,
"max" : 95610,
"mean" : 9103.494547,
"stddev" : 2144.724722,
"N" : 620719,
"percentile" : {
"1.000000" : 8512,
"5.000000" : 8640,
"10.000000" : 8640,
"20.000000" : 8640,
"30.000000" : 8768,
"40.000000" : 8768,
"50.000000" : 8768,
"60.000000" : 8768,
"70.000000" : 8768,
"80.000000" : 8896,
"90.000000" : 9024,
"95.000000" : 9536,
"99.000000" : 19328,
"99.500000" : 24960,
"99.900000" : 26240,
"99.950000" : 41728,
"99.990000" : 65280
}
},
"lat_ns" : {
"min" : 9959,
"max" : 97583,
"mean" : 11030.278872,
"stddev" : 2148.473123,
"N" : 620719
},
"bw_min" : 353864,
"bw_max" : 355863,
"bw_agg" : 100.000000,
"bw_mean" : 354830.928571,
"bw_dev" : 468.121198,
"bw_samples" : 14,
"iops_min" : 88466,
"iops_max" : 88965,
"iops_mean" : 88707.642857,
"iops_stddev" : 116.917201,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 34.157143,
"sys_cpu" : 65.814286,
"ctx" : 19,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 95.638445,
"20" : 3.521722,
"50" : 0.793274,
"100" : 0.039309,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=85581: Tue Oct 25 23:29:49 2022
read: IOPS=88.7k, BW=346MiB/s (363MB/s)(2425MiB/7001msec)
slat (nsec): min=1783, max=14928, avg=1926.78, stdev=147.91
clat (nsec): min=611, max=95610, avg=9103.49, stdev=2144.72
lat (nsec): min=9959, max=97583, avg=11030.28, stdev=2148.47
clat percentiles (nsec):
| 1.00th=[ 8512], 5.00th=[ 8640], 10.00th=[ 8640], 20.00th=[ 8640],
| 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8768], 60.00th=[ 8768],
| 70.00th=[ 8768], 80.00th=[ 8896], 90.00th=[ 9024], 95.00th=[ 9536],
| 99.00th=[19328], 99.50th=[24960], 99.90th=[26240], 99.95th=[41728],
| 99.99th=[65280]
bw ( KiB/s): min=353864, max=355863, per=100.00%, avg=354830.93, stdev=468.12, samples=14
iops : min=88466, max=88965, avg=88707.64, stdev=116.92, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 4=0.01%, 10=95.64%, 20=3.52%, 50=0.79%, 100=0.04%
cpu : usr=34.16%, sys=65.81%, ctx=19, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=620719,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=346MiB/s (363MB/s), 346MiB/s-346MiB/s (363MB/s-363MB/s), io=2425MiB (2542MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733404,
"timestamp_ms" : 1666733404980,
"time" : "Tue Oct 25 23:30:04 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2823823360,
"io_kbytes" : 2757640,
"bw_bytes" : 403403337,
"bw" : 393948,
"iops" : 98487.142857,
"runtime" : 7000,
"total_ios" : 689410,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1582,
"max" : 36869,
"mean" : 1652.357284,
"stddev" : 116.694315,
"N" : 689410
},
"clat_ns" : {
"min" : 651,
"max" : 112781,
"mean" : 8251.688869,
"stddev" : 1885.218026,
"N" : 689410,
"percentile" : {
"1.000000" : 7776,
"5.000000" : 7776,
"10.000000" : 7776,
"20.000000" : 7840,
"30.000000" : 7840,
"40.000000" : 7904,
"50.000000" : 7904,
"60.000000" : 7904,
"70.000000" : 8032,
"80.000000" : 8160,
"90.000000" : 8256,
"95.000000" : 8640,
"99.000000" : 17280,
"99.500000" : 19328,
"99.900000" : 22656,
"99.950000" : 41216,
"99.990000" : 67072
}
},
"lat_ns" : {
"min" : 9198,
"max" : 114405,
"mean" : 9904.046153,
"stddev" : 1889.202305,
"N" : 689410
},
"bw_min" : 393288,
"bw_max" : 394592,
"bw_agg" : 100.000000,
"bw_mean" : 393990.571429,
"bw_dev" : 426.285876,
"bw_samples" : 14,
"iops_min" : 98322,
"iops_max" : 98648,
"iops_mean" : 98497.642857,
"iops_stddev" : 106.571469,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 33.642857,
"sys_cpu" : 66.328571,
"ctx" : 17,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.026167,
"20" : 3.730146,
"50" : 0.206408,
"100" : 0.035683,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=85630: Tue Oct 25 23:30:04 2022
read: IOPS=98.5k, BW=385MiB/s (403MB/s)(2693MiB/7000msec)
slat (nsec): min=1582, max=36869, avg=1652.36, stdev=116.69
clat (nsec): min=651, max=112781, avg=8251.69, stdev=1885.22
lat (usec): min=9, max=114, avg= 9.90, stdev= 1.89
clat percentiles (nsec):
| 1.00th=[ 7776], 5.00th=[ 7776], 10.00th=[ 7776], 20.00th=[ 7840],
| 30.00th=[ 7840], 40.00th=[ 7904], 50.00th=[ 7904], 60.00th=[ 7904],
| 70.00th=[ 8032], 80.00th=[ 8160], 90.00th=[ 8256], 95.00th=[ 8640],
| 99.00th=[17280], 99.50th=[19328], 99.90th=[22656], 99.95th=[41216],
| 99.99th=[67072]
bw ( KiB/s): min=393288, max=394592, per=100.00%, avg=393990.57, stdev=426.29, samples=14
iops : min=98322, max=98648, avg=98497.64, stdev=106.57, samples=14
lat (nsec) : 750=0.01%
lat (usec) : 4=0.01%, 10=96.03%, 20=3.73%, 50=0.21%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=33.64%, sys=66.33%, ctx=17, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=689410,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=385MiB/s (403MB/s), 385MiB/s-385MiB/s (403MB/s-403MB/s), io=2693MiB (2824MB), run=7000-7000msec
$ rm /tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733420,
"timestamp_ms" : 1666733420440,
"time" : "Tue Oct 25 23:30:20 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1"
},
"read" : {
"io_bytes" : 2837221376,
"io_kbytes" : 2770724,
"bw_bytes" : 405259445,
"bw" : 395761,
"iops" : 98940.294244,
"runtime" : 7001,
"total_ios" : 692681,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1563,
"max" : 52007,
"mean" : 1627.296386,
"stddev" : 124.623637,
"N" : 692681
},
"clat_ns" : {
"min" : 631,
"max" : 93886,
"mean" : 8236.372760,
"stddev" : 1870.436580,
"N" : 692681,
"percentile" : {
"1.000000" : 7584,
"5.000000" : 7648,
"10.000000" : 7840,
"20.000000" : 7904,
"30.000000" : 7904,
"40.000000" : 7904,
"50.000000" : 7968,
"60.000000" : 7968,
"70.000000" : 7968,
"80.000000" : 7968,
"90.000000" : 8032,
"95.000000" : 8640,
"99.000000" : 17280,
"99.500000" : 19328,
"99.900000" : 22656,
"99.950000" : 40704,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 9167,
"max" : 95499,
"mean" : 9863.669146,
"stddev" : 1874.856856,
"N" : 692681
},
"bw_min" : 395336,
"bw_max" : 396672,
"bw_agg" : 100.000000,
"bw_mean" : 395852.500000,
"bw_dev" : 386.086230,
"bw_samples" : 14,
"iops_min" : 98834,
"iops_max" : 99168,
"iops_mean" : 98963.071429,
"iops_stddev" : 96.554858,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 28.842857,
"sys_cpu" : 71.128571,
"ctx" : 44,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.036992,
"20" : 3.718595,
"50" : 0.209476,
"100" : 0.033926,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=85679: Tue Oct 25 23:30:20 2022
read: IOPS=98.9k, BW=386MiB/s (405MB/s)(2706MiB/7001msec)
slat (nsec): min=1563, max=52007, avg=1627.30, stdev=124.62
clat (nsec): min=631, max=93886, avg=8236.37, stdev=1870.44
lat (nsec): min=9167, max=95499, avg=9863.67, stdev=1874.86
clat percentiles (nsec):
| 1.00th=[ 7584], 5.00th=[ 7648], 10.00th=[ 7840], 20.00th=[ 7904],
| 30.00th=[ 7904], 40.00th=[ 7904], 50.00th=[ 7968], 60.00th=[ 7968],
| 70.00th=[ 7968], 80.00th=[ 7968], 90.00th=[ 8032], 95.00th=[ 8640],
| 99.00th=[17280], 99.50th=[19328], 99.90th=[22656], 99.95th=[40704],
| 99.99th=[66048]
bw ( KiB/s): min=395336, max=396672, per=100.00%, avg=395852.50, stdev=386.09, samples=14
iops : min=98834, max=99168, avg=98963.07, stdev=96.55, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 10=96.04%, 20=3.72%, 50=0.21%, 100=0.03%
cpu : usr=28.84%, sys=71.13%, ctx=44, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=692681,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=386MiB/s (405MB/s), 386MiB/s-386MiB/s (405MB/s-405MB/s), io=2706MiB (2837MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-xnvme_0.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-xnvme_0.txt"
$ cat /tmp/fio-output_4K_1_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733434,
"timestamp_ms" : 1666733434249,
"time" : "Tue Oct 25 23:30:34 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 1678741504,
"io_kbytes" : 1639396,
"bw_bytes" : 239785959,
"bw" : 234165,
"iops" : 58541.494072,
"runtime" : 7001,
"total_ios" : 409849,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2966,
"max" : 24035,
"mean" : 3340.552930,
"stddev" : 206.631663,
"N" : 409849
},
"clat_ns" : {
"min" : 1123,
"max" : 438051,
"mean" : 13173.901883,
"stddev" : 2319.222617,
"N" : 409849,
"percentile" : {
"1.000000" : 11968,
"5.000000" : 12096,
"10.000000" : 12096,
"20.000000" : 12224,
"30.000000" : 12352,
"40.000000" : 12480,
"50.000000" : 13376,
"60.000000" : 13376,
"70.000000" : 13376,
"80.000000" : 13504,
"90.000000" : 13504,
"95.000000" : 13632,
"99.000000" : 23680,
"99.500000" : 28800,
"99.900000" : 35072,
"99.950000" : 50944,
"99.990000" : 73216
}
},
"lat_ns" : {
"min" : 12203,
"max" : 441297,
"mean" : 16514.454814,
"stddev" : 2343.548276,
"N" : 409849
},
"bw_min" : 226544,
"bw_max" : 245707,
"bw_agg" : 100.000000,
"bw_mean" : 234225.357143,
"bw_dev" : 8659.960984,
"bw_samples" : 14,
"iops_min" : 56636,
"iops_max" : 61426,
"iops_mean" : 58556.285714,
"iops_stddev" : 2164.913764,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 12.657143,
"sys_cpu" : 36.671429,
"ctx" : 409826,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 98.453089,
"50" : 1.489085,
"100" : 0.050506,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=85718: Tue Oct 25 23:30:34 2022
read: IOPS=58.5k, BW=229MiB/s (240MB/s)(1601MiB/7001msec)
slat (nsec): min=2966, max=24035, avg=3340.55, stdev=206.63
clat (nsec): min=1123, max=438051, avg=13173.90, stdev=2319.22
lat (usec): min=12, max=441, avg=16.51, stdev= 2.34
clat percentiles (nsec):
| 1.00th=[11968], 5.00th=[12096], 10.00th=[12096], 20.00th=[12224],
| 30.00th=[12352], 40.00th=[12480], 50.00th=[13376], 60.00th=[13376],
| 70.00th=[13376], 80.00th=[13504], 90.00th=[13504], 95.00th=[13632],
| 99.00th=[23680], 99.50th=[28800], 99.90th=[35072], 99.95th=[50944],
| 99.99th=[73216]
bw ( KiB/s): min=226544, max=245707, per=100.00%, avg=234225.36, stdev=8659.96, samples=14
iops : min=56636, max=61426, avg=58556.29, stdev=2164.91, samples=14
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=98.45%, 50=1.49%
lat (usec) : 100=0.05%, 250=0.01%, 500=0.01%
cpu : usr=12.66%, sys=36.67%, ctx=409826, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=409849,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=1601MiB (1679MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-xnvme_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-xnvme_1.txt"
$ cat /tmp/fio-output_4K_1_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733448,
"timestamp_ms" : 1666733448033,
"time" : "Tue Oct 25 23:30:48 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 1764671488,
"io_kbytes" : 1723312,
"bw_bytes" : 252059918,
"bw" : 246152,
"iops" : 61538.065991,
"runtime" : 7001,
"total_ios" : 430828,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2925,
"max" : 175278,
"mean" : 3159.793955,
"stddev" : 387.428894,
"N" : 430828
},
"clat_ns" : {
"min" : 1002,
"max" : 182192,
"mean" : 12504.355956,
"stddev" : 2223.696600,
"N" : 430828,
"percentile" : {
"1.000000" : 11968,
"5.000000" : 11968,
"10.000000" : 12096,
"20.000000" : 12096,
"30.000000" : 12096,
"40.000000" : 12224,
"50.000000" : 12224,
"60.000000" : 12224,
"70.000000" : 12352,
"80.000000" : 12352,
"90.000000" : 12480,
"95.000000" : 12608,
"99.000000" : 22144,
"99.500000" : 29056,
"99.900000" : 37632,
"99.950000" : 52992,
"99.990000" : 73216
}
},
"lat_ns" : {
"min" : 12213,
"max" : 188343,
"mean" : 15664.149911,
"stddev" : 2259.446273,
"N" : 430828
},
"bw_min" : 245008,
"bw_max" : 247408,
"bw_agg" : 100.000000,
"bw_mean" : 246227.357143,
"bw_dev" : 707.156343,
"bw_samples" : 14,
"iops_min" : 61252,
"iops_max" : 61852,
"iops_mean" : 61556.785714,
"iops_stddev" : 176.840727,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.242857,
"sys_cpu" : 39.485714,
"ctx" : 430779,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.010213,
"20" : 98.469227,
"50" : 1.455801,
"100" : 0.060581,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=85758: Tue Oct 25 23:30:48 2022
read: IOPS=61.5k, BW=240MiB/s (252MB/s)(1683MiB/7001msec)
slat (nsec): min=2925, max=175278, avg=3159.79, stdev=387.43
clat (nsec): min=1002, max=182192, avg=12504.36, stdev=2223.70
lat (usec): min=12, max=188, avg=15.66, stdev= 2.26
clat percentiles (nsec):
| 1.00th=[11968], 5.00th=[11968], 10.00th=[12096], 20.00th=[12096],
| 30.00th=[12096], 40.00th=[12224], 50.00th=[12224], 60.00th=[12224],
| 70.00th=[12352], 80.00th=[12352], 90.00th=[12480], 95.00th=[12608],
| 99.00th=[22144], 99.50th=[29056], 99.90th=[37632], 99.95th=[52992],
| 99.99th=[73216]
bw ( KiB/s): min=245008, max=247408, per=100.00%, avg=246227.36, stdev=707.16, samples=14
iops : min=61252, max=61852, avg=61556.79, stdev=176.84, samples=14
lat (usec) : 2=0.01%, 10=0.01%, 20=98.47%, 50=1.46%, 100=0.06%
lat (usec) : 250=0.01%
cpu : usr=13.24%, sys=39.49%, ctx=430779, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=430828,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=240MiB/s (252MB/s), 240MiB/s-240MiB/s (252MB/s-252MB/s), io=1683MiB (1765MB), run=7001-7001msec
$ rm /tmp/fio-output_4K_1_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-xnvme_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-xnvme_2.txt"
$ cat /tmp/fio-output_4K_1_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666733461,
"timestamp_ms" : 1666733461706,
"time" : "Tue Oct 25 23:31:01 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "4K",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1"
},
"read" : {
"io_bytes" : 1754910720,
"io_kbytes" : 1713780,
"bw_bytes" : 250665722,
"bw" : 244790,
"iops" : 61197.686045,
"runtime" : 7001,
"total_ios" : 428445,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2905,
"max" : 25257,
"mean" : 3138.958107,
"stddev" : 264.649411,
"N" : 428445
},
"clat_ns" : {
"min" : 1072,
"max" : 196789,
"mean" : 12600.985377,
"stddev" : 2183.473992,
"N" : 428445,
"percentile" : {
"1.000000" : 11968,
"5.000000" : 12096,
"10.000000" : 12096,
"20.000000" : 12224,
"30.000000" : 12224,
"40.000000" : 12224,
"50.000000" : 12352,
"60.000000" : 12352,
"70.000000" : 12480,
"80.000000" : 12480,
"90.000000" : 12480,
"95.000000" : 12736,
"99.000000" : 22400,
"99.500000" : 29312,
"99.900000" : 32384,
"99.950000" : 50944,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 12002,
"max" : 199895,
"mean" : 15739.943484,
"stddev" : 2203.074385,
"N" : 428445
},
"bw_min" : 243672,
"bw_max" : 246248,
"bw_agg" : 100.000000,
"bw_mean" : 244847.571429,
"bw_dev" : 682.736655,
"bw_samples" : 14,
"iops_min" : 60918,
"iops_max" : 61562,
"iops_mean" : 61211.857143,
"iops_stddev" : 170.666502,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 12.700000,
"sys_cpu" : 39.757143,
"ctx" : 428394,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 98.445308,
"50" : 1.491907,
"100" : 0.050415,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=85798: Tue Oct 25 23:31:01 2022
read: IOPS=61.2k, BW=239MiB/s (251MB/s)(1674MiB/7001msec)
slat (nsec): min=2905, max=25257, avg=3138.96, stdev=264.65
clat (nsec): min=1072, max=196789, avg=12600.99, stdev=2183.47
lat (usec): min=12, max=199, avg=15.74, stdev= 2.20
clat percentiles (nsec):
| 1.00th=[11968], 5.00th=[12096], 10.00th=[12096], 20.00th=[12224],
| 30.00th=[12224], 40.00th=[12224], 50.00th=[12352], 60.00th=[12352],
| 70.00th=[12480], 80.00th=[12480], 90.00th=[12480], 95.00th=[12736],
| 99.00th=[22400], 99.50th=[29312], 99.90th=[32384], 99.95th=[50944],
| 99.99th=[72192]
bw ( KiB/s): min=243672, max=246248, per=100.00%, avg=244847.57, stdev=682.74, samples=14
iops : min=60918, max=61562, avg=61211.86, stdev=170.67, samples=14
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=98.45%, 50=1.49%
lat (usec) : 100=0.05%, 250=0.01%
cpu : usr=12.70%, sys=39.76%, ctx=428394, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=428445,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=239MiB/s (251MB/s), 239MiB/s-239MiB/s (251MB/s-251MB/s), io=1674MiB (1755MB), run=7001-7001msec
Steps uses worklets. A description of the worklet used by
the current step is provided below. The description is extracted from the docstring
of the worklet and provided here in verbatim / as-is.
Extract and transform fio and bdevperf output
=============================================
fio has:
* bandwidth in KiB / second
* latency is mean and in nano-seconds
* IOPS are 'raw' / base-unit
bdevperf has:
* bandwidth in MiB / second
* IOPS are 'raw' / base-unit
NOTE: The metric-context need addition of backend-options, the options are
semi-encoded by ioengine and 'label', however, that is not very precise.
{'description': '\n'
' Extract and transform fio and bdevperf output\n'
' =============================================\n'
'\n'
' fio has:\n'
'\n'
' * bandwidth in KiB / second\n'
' * latency is mean and in nano-seconds\n'
" * IOPS are 'raw' / base-unit\n"
'\n'
' bdevperf has:\n'
'\n'
' * bandwidth in MiB / second\n'
" * IOPS are 'raw' / base-unit\n"
'\n'
' NOTE: The metric-context need addition of '
'backend-options, the options are\n'
" semi-encoded by ioengine and 'label', however, that is "
'not very precise.\n',
'extras': {},
'id': '003_plot_fio',
'name': 'plot_fio',
'nr': 3,
'status': {'elapsed': 0.7708134651184082,
'failed': 0,
'passed': 1,
'skipped': 0,
'started': 0.0},
'uses': 'plot',
'with': {'path': 'cijoe-output', 'tool': 'fio'}}
Steps uses worklets. A description of the worklet used by
the current step is provided below. The description is extracted from the docstring
of the worklet and provided here in verbatim / as-is.
Extract and transform fio and bdevperf output
=============================================
fio has:
* bandwidth in KiB / second
* latency is mean and in nano-seconds
* IOPS are 'raw' / base-unit
bdevperf has:
* bandwidth in MiB / second
* IOPS are 'raw' / base-unit
NOTE: The metric-context need addition of backend-options, the options are
semi-encoded by ioengine and 'label', however, that is not very precise.
{'description': '\n'
' Extract and transform fio and bdevperf output\n'
' =============================================\n'
'\n'
' fio has:\n'
'\n'
' * bandwidth in KiB / second\n'
' * latency is mean and in nano-seconds\n'
" * IOPS are 'raw' / base-unit\n"
'\n'
' bdevperf has:\n'
'\n'
' * bandwidth in MiB / second\n'
" * IOPS are 'raw' / base-unit\n"
'\n'
' NOTE: The metric-context need addition of '
'backend-options, the options are\n'
" semi-encoded by ioengine and 'label', however, that is "
'not very precise.\n',
'extras': {},
'id': '004_plot_bdevperf',
'name': 'plot_bdevperf',
'nr': 4,
'status': {'elapsed': 0.48505663871765137,
'failed': 0,
'passed': 1,
'skipped': 0,
'started': 0.0},
'uses': 'plot',
'with': {'path': 'cijoe-output', 'tool': 'bdevperf'}}